Top

Published in:

2019 | OriginalPaper | Chapter

Polyhedral Optimization of TensorFlow Computation Graphs

Authors : Benoît Pradelle, Benoît Meister, Muthu Baskaran, Jonathan Springer, Richard Lethin

Published in: Programming and Performance Visualization Tools

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

We present \({\textsf {R}}\text {-}{\textsf {Stream}}{\cdot }{\textsf {TF}}\), a polyhedral optimization tool for neural network computations. \({\textsf {R}}\text {-}{\textsf {Stream}}{\cdot }{\textsf {TF}}\) transforms computations performed in a neural network graph into C programs suited to the polyhedral representation and uses R-Stream, a polyhedral compiler, to parallelize and optimize the computations performed in the graph. \({\textsf {R}}\text {-}{\textsf {Stream}}{\cdot }{\textsf {TF}}\) can exploit the optimizations available with R-Stream to generate a highly optimized version of the computation graph, specifically mapped to the targeted architecture. During our experiments, \({\textsf {R}}\text {-}{\textsf {Stream}}{\cdot }{\textsf {TF}}\) was able to automatically reach performance levels close to the hand-optimized implementations, demonstrating its utility in porting neural network computations to parallel architectures.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Moya—A JIT Compiler for HPC

next chapter CAASCADE: A System for Static Analysis of HPC Software Application Portfolios

Raising generic C codes into a polyhedral representation can be a complex problem, when programmers or library writers have made manual optimizations (e.g., parallelization, tiling, ...) based on domain knowledge which cannot be easily inferred from their program source. Such manual optimizations are often not performance portable (or portable at all) to new platforms, thus their action of performing manual optimization “bakes the code” to that one original target. To re-optimize to a new architecture through the polyhedral model, such manual optimizations often have to be reverted to produce an efficient polyhedral representation of the program. Unknown aliasing and overflowing arithmetic are among the challenges of such “un-baking.” With modern compiler tools like R-Stream now available, it would a much more sustainable practice for programmers to express their code originally in a high-level, domain-specific manner.

Khandekar, R., et al.: COLA: optimizing stream processing applications via graph partitioning. In: Bacon, J.M., Cooper, B.F. (eds.) Middleware 2009. LNCS, vol. 5896, pp. 308–327. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10445-9_16CrossRef

Abadi, M., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems (2015). http://download.tensorflow.org/paper/whitepaper2015.pdf

Atasu, K., Pozzi, L., Ienne, P.: Automatic application-specific instruction-set extensions under microarchitectural constraints. In: Proceedings of the 40th Annual Design Automation Conference, pp. 256–261. ACM (2003)

Baghdadi, R., et al.: PENCIL: a platform-neutral compute intermediate language for accelerator programming. In: Parallel Architectures and Compilation Techniques (PACT), San Francisco, CA, USA (2015). https://hal.archives-ouvertes.fr/hal-01257236

Bastoul, C., et al.: System, methods and apparatus for program optimization for multi-threaded processor architectures, April 2010

Bergstra, J., et al.: Theano: a CPU and GPU math compiler in Python (2011)

Bondhugula, U., Baskaran, M., Krishnamoorthy, S., Ramanujam, J., Rountev, A., Sadayappan, P.: Automatic transformations for communication-minimized parallelization and locality optimization in the polyhedral model. In: Hendren, L. (ed.) CC 2008. LNCS, vol. 4959, pp. 132–146. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78791-4_9CrossRef

Chen, T., et al.: MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. CoRR abs/1512.01274 (2015). http://arxiv.org/abs/1512.01274

Clauss, P., Meister, B.: Automatic memory layout transformation to optimize spatial locality in parameterized loop nests. ACM SIGARCH Comput. Archit. News 28(1), 11–19 (2000)CrossRef

10.

Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: a matlab-like environment for machine learning. In: BigLearn, NIPS Workshop No. EPFL-CONF-192376 (2011)

11.

Dayarathna, M., Suzumura, T.: Automatic optimization of stream programs via source program operator graph transformations. Distrib. Parallel Databases 31(4), 543–599 (2013). https://doi.org/10.1007/s10619-013-7130-xCrossRef

12.

Feautrier, P.: Some efficient solutions to the affine scheduling problem. Part I. One-dimensional time. Int. J. Parallel Program. 21(5), 313–348 (1992). citeseer.ist.psu.edu/feautrier92some.html

13.

Grosser, T., Groesslinger, A., Lengauer, C.: Polly-performing polyhedral optimizations on a low-level intermediate representation. Parallel Process. Lett. 22(04), 1250010 (2012)MathSciNetCrossRef

14.

Irigoin, F., Triolet, R.: Supernode partitioning. In: Proceedings of the 15th ACM SIGPLAN-SIGACT Symposium on Principles of programming languages, pp. 319–329. ACM Press, New York, January 1988

15.

Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding (2014)

16.

LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRef

17.

Lethin, R.A., Leung, A.K., Meister, B.J., Vasilache, N.T.: Methods and apparatus for joint parallelism and locality optimization in source code compilation, September 2009

18.

Meister, B., Baskaran, M.M., Pradelle, B., Henretty, T., Lethin, R.: Efficient compilation to event-driven task programs. CoRR abs/1601.05458 (2016). http://arxiv.org/abs/1601.05458

19.

Meister, B., Vasilache, N., Wohlford, D., Baskaran, M.M., Leung, A., Lethin, R.: R-stream compiler. In: Padua, D. (ed.) Encyclopedia of Parallel Computing, pp. 1756–1765. Springer, Boston (2011). https://doi.org/10.1007/978-0-387-09766-4CrossRef

20.

Meister, B., et al.: SANE: an array language for sensor applications. In: Proceedings of a Workshop on Domain-Specific Languages and High-Level Frameworks for High Performance Computing, Salt Lake City, UT, USA, November 16 2012

21.

Pop, S., et al.: GRAPHITE: loop optimizations based on the polyhedral model for GCC. In: Proceedings of the 4th GCC Developper’s Summit, pp. 179–198. Ottawa, Canada, June 2006

22.

Pradelle, B., Meister, B., Baskaran, M.M., Konstantinidis, A., Henretty, T., Lethin, R.: Scalable hierarchical polyhedral compilation. In: International Conference on Parallel Processing (ICPP) (2016)

23.

Vasilache, N., et al.: A tale of three runtimes. arXiv:1409.1914

24.

Verdoolaege, S.: isl: an integer set library for the polyhedral model. In: Proceedings of the Third international congress conference on Mathematical software (ICMS 2010), pp. 299–302. ACM Press (2010)

25.

Verdoolaege, S., Janssens, G.: Scheduling for PPCG. Technical report, Department of Computer Science, KU Leuven (2017)

26.

Verdoolaege, S., Juega, J.C., Cohen, A., Gómez, J.I., Tenllado, C., Catthoor, F.: Polyhedral parallel code generation for CUDA. ACM Trans. Archit. Code Optim. 9(4), 54:1–54:23 (2013)CrossRef

27.

Yu, D., Yao, K., Zhang, Y.: The computational network toolkit [best of the web]. IEEE Signal Process. Mag. 32(6), 123–126 (2015)CrossRef

Title: Polyhedral Optimization of TensorFlow Computation Graphs
Authors: Benoît Pradelle
Benoît Meister
Muthu Baskaran
Jonathan Springer
Richard Lethin
Publisher: Springer International Publishing
Book: Programming and Performance Visualization Tools
Print ISBN: 978-3-030-17871-0

Electronic ISBN: 978-3-030-17872-7

Copyright Year: 2019
DOI: https://doi.org/10.1007/978-3-030-17872-7_5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner