Roy Schwartz
Skip Abstract Section
Abstract
Creating efficiency in AI research will decrease its carbon footprint and increase its inclusivity as deep learning study should not require the deepest pockets.
- Acharyya, P., Rosario, S.D., Flor, F., Joshi, R., Li, D., Linares, R, and Zhang, H. Autopilot of cement plants for reduction of fuel consumption and emissions. In Proceedings of ICML Workshop on Climate Change, 2019.Google Scholar
- Amodei, D. and Hernandez, D. AI and compute, 2018. Blog post.Google Scholar
- Bergstra, J.S., Bardenet, R., Bengio, Y. and Kégl, B. Algorithms for hyper-parameter optimization. In Proceedings of NeurIPS, 2011.Google Scholar
- Brown, T.B. et al. Language models are few-shot learners, 2020; arXiv:2005.14165.Google Scholar
- Canziani, A., Paszke, A. and Culurciello, E. An analysis of deep neural network models for practical applications. In Proceedings of ISCAS, 2017.Google Scholar
- Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S. and Feng, J. Dual path networks. In Proceedings of NeurIPS, 2017.Google Scholar
- Deng, J., Dong, W., Socher, R., Li, L-J, Li, K. and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proceedings of CVPR, 2009.Google ScholarCross Ref
- Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. BERT: Pretraining of deep bidirectional transformers for language understanding. In Proceedings of NAACL, 2019.Google Scholar
- Dodge, J., Gururangan, S., Card, D., Schwartz, R. and Smith, N.A. Show your work: Improved reporting of experimental results. In Proceedings of EMNLP, 2019.Google ScholarCross Ref
- Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H. and Smith, N.A. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping, 2020; arXiv:2002.06305.Google Scholar
- Dodge, J., Jamieson, K. and Smith, N.A. Open loop hyperparameter optimization and determinantal point processes. In Proceedings of AutoML, 2017.Google Scholar
- Duhart, C., Dublon, G., Mayton, B., Davenport, G. and Paradiso, J.A. Deep learning for wildlife conservation and restoration efforts. In Proceedings of ICML Workshop on Climate Change, 2019.Google Scholar
- Gordon, A., Eban, E., Nachum, O., Chen, B., Wu, H., Yang, T-J, and Choi, E. MorphNet: Fast & simple resource-constrained structure learning of deep networks. In Proceedings of CVPR, 2018.Google ScholarCross Ref
- Halevy, A., Norvig, P. and Pereira, F. The unreasonable effectiveness of data. IEEE Intelligent Systems 24 (2009), 8--12.Google ScholarDigital Library
- He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. In Proceedings of CVPR, 2016.Google ScholarCross Ref
- Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D. and Pineau, J. Towards the systematic reporting of the energy and carbon footprints of machine learning, 2020; arXiv:2002.05651.Google Scholar
- Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation 9, 8 (1997), 1735--1780.Google ScholarDigital Library
- Howard, A.G. et al. MobileNets: Efficient convolutional neural networks for mobile vision applications, 2017; arXiv:1704.04861.Google Scholar
- Hu, J., Shen, L. and Sun, G. Squeeze-and-excitation networks. In Proceedings of CVPR, 2018.Google ScholarCross Ref
- Huang, J. et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of CVPR, 2017.Google ScholarCross Ref
- Jeon, Y. and Kim, J. Constructing fast network through deconstruction of convolution. In Proceedings of NeurIPS, 2018.Google Scholar
- Jouppi, N.P. et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of ISCA 1, 1 (2017), Publ. date: June 2020.Google Scholar
- Kamthe, S. and Deisenroth, M.P. Data-efficient reinforcement learning with probabilistic model predictive control. In Proceedings of AISTATS, 2018.Google Scholar
- Krizhevsky, A., Sutskever, I. and Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Proceedings of NeurIPS, 2012.Google Scholar
- Lacoste, A., Luccioni, A., Schmidt, V. and Dandres, T. Quantifying the carbon emissions of machine learning. In Proceedings of the Climate Change AI Workshop, 2019.Google Scholar
- Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A. and Talwalkar, A. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In Proceedings of ICLR, 2017.Google Scholar
- Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S. Fu, C-Y and Berg, A.C. SSD: Single shot multibox detector. In Proceedings of ECCV, 2016.Google ScholarCross Ref
- Liu, Y. et al. RoBERTa: A robustly optimized BERT pretraining approach, 2019; arXiv:1907.11692.Google Scholar
- Ma, N., Zhang, X., Zheng, H.T and Sun, J. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In Proceedings of ECCV, 2018.Google ScholarCross Ref
- Mahajan, D. et al. Exploring the limits of weakly supervised pretraining, 2018; arXiv:1805.00932.Google Scholar
- Melis, G., Dyer, C. and Blunsom, P. On the state of the art of evaluation in neural language models. In Proceedings of EMNLP, 2018.Google Scholar
- Molchanov, P., Tyree, S., Karras, T., Aila, T. and Kautz, J. Pruning convolutional neural networks for resource efficient inference. In Proceedings of ICLR, 2017.Google Scholar
- Moore, G.E. Cramming more components onto integrated circuits, 1965.Google Scholar
- Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K. and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of NAACL, 2018.Google ScholarCross Ref
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. Language models are unsupervised multitask learners.. OpenAI Blog, 2019.Google Scholar
- Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019; arXiv:1910.10683.Google Scholar
- Rastegari, M., Ordonez, V., Redmon, J. and Farhadi, A. Xnornet: Imagenet classification using binary convolutional neural networks. In Proceedings of ECCV, 2016.Google ScholarCross Ref
- Redmon, J., Divvala, S., Girshick, R. and Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of CVPR, 2016.Google ScholarCross Ref
- Rolnick, D. et al. Tackling climate change with machine learning, 2019; arXiv:1905.12616.Google Scholar
- Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of CVPR, 2018.Google ScholarCross Ref
- Schwartz, R., Thomson, S. and Smith, N.A. SoPa: Bridging CNNs, RNNs, and weighted finite-state machines. In Proceedings of ACL, 2018.Google Scholar
- Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B. Megatron-LM: Training multi-billion parameter language models using GPU model parallelism, 2019; arXiv:1909.08053.Google Scholar
- Shoham, Y. et al. The AI index 2018 annual report. AI Index Steering Committee, Human-Centered AI Initiative, Stanford University; http://cdn.aiindex.org/2018/AI%20Index%202018%20Annual%20Report.pdf.Google Scholar
- Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016) 484.Google ScholarCross Ref
- Silver, D. et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017; arXiv:1712.01815.Google Scholar
- Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 7676 (2017), 354.Google Scholar
- Strubell, E., Ganesh, A. and McCallum, A. Energy and policy considerations for deep learning in NLP. In Proceedings of ACL, 2019.Google ScholarCross Ref
- Sun, C., Shrivastava, A., Singh, S. and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of ICCV, 2017.Google ScholarCross Ref
- Tsang, I., Kwok, J.T. and Cheung, P.M. Core vector machines: Fast SVM training on very large data sets. JMLR 6 (Apr. 2005), 363--392.Google Scholar
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. Attention is all you need. In Proceedings of NeurIPS, 2017.Google Scholar
- Veniat, T. and Denoyer, L. Learning time/memory-efficient deep architectures with budgeted super networks. In Proceedings of CVPR, 2018.Google ScholarCross Ref
- Walsman, A., Bisk, Y., Gabriel, S., Misra, D., Artzi, Y., Choi, Y. and Fox, D. Early fusion for goal directed robotic vision. In Proceedings of IROS, 2019.Google ScholarDigital Library
- Wang, A. Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O. and Bowman, S.R. SuperGLUE: A stickier benchmark for general-purpose language understanding systems, 2019; arXiv:1905.00537.Google Scholar
- Wang, A., Singh, A., Michael, J., Hill, F., Levy, O. and Bowman, S.R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR, 2019.Google Scholar
- Xie, S., Girshick, R., Dollar, P., Tu, Z. and He, K. Aggregated residual transformations for deep neural networks. In Proceedings of CVPR, 2017.Google ScholarCross Ref
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. and Le, Q.V. XLNet: Generalized autoregressive pretraining for language understanding, 2019; arXiv:1906.08237.Google Scholar
- Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F. and Choi, Y. Defending against neural fake news, 2019; arXiv:1905.12616.Google Scholar
- Zhang, X., Zhou, X., Lin, M. and Sun, J. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of CVPR, 2018.Google ScholarCross Ref
- Zoph, B. and Le, Q.V. Neural architecture search with reinforcement learning. In Proceedings of ICLR, 2017.Google Scholar
Index Terms
- Green AI
Recommendations
Harnessing Green IT: Principles and Practices
Adopting a holistic approach to greening IT is our responsibility toward creating a more sustaining environment.
Go Green with WEEE: Eco-friendly Approach for Handling E- waste
AbstractThe rate of EEE put in market and WEEE generated in previous years defines the magnitude of the problem of e-waste handling. Many researches have indicated the potential threat of e-waste and stated the need of proper recycling strategies and ...
Making IT Green
IT is both a solution and problem for environmental sustainability. As businesses and governments try to balance growth with environmental risks, we're called upon to make IT systems and work practices greener and to use IT in innovative ways to address ...
Comments