Dimitri Komatitsch
ABSTRACT
We use 1944 processors of the Earth Simulator to model seismic wave propagation resulting from large earthquakes. Simulations are conducted based upon the spectral-element method, a high-degree finite-element technique with an exactly diagonal mass matrix. We use a very large mesh with 5.5 billion grid points (14.6 billion degrees of freedom). We include the full complexity of the Earth, i.e., a three-dimensional wave-speed and density structure, a 3-D crustal model, ellipticity as well as topography and bathymetry. A total of 2.5 terabytes of memory is needed. Our implementation is purely based upon MPI, with loop vectorization on each processor. We obtain an excellent vectorization ratio of 99.3%, and we reach a performance of 5 teraflops (30% of the peak performance) on 38% of the machine. The very high resolution of the mesh allows us to perform fully three-dimensional calculations at seismic periods as low as 5 seconds.
- {1} C. Bassin, G. Laske, and G. Masters. The current limits of resolution for surface wave tomography in North America. EOS, 81:F897, 2000.Google Scholar
- {2} E. Chaljub. Modélisation numérique de la propagation d'ondes sismiques en géométrie sphérique: application à la sismologie globale (Numerical modeling of the propagation of seismic waves in spherical geometry: applications to global seismology). PhD thesis, Université Paris VII Denis Diderot, Paris, France, 2000.Google Scholar
- {3} F. A. Dahlen and J. Tromp. Theoretical Global Seismology. Princeton University Press, Princeton, 1998.Google Scholar
- {4} A. M. Dziewonski and D. L. Anderson. Preliminary reference Earth model. Phys. Earth Planet. Inter., 25:297-356, 1981.Google ScholarCross Ref
- {5} D. Eberhart-Phillips, P. J. Haeussler, J. T. Freymueller, A. D. Frankel, C. M. Rubin, P. Craw, N. A. Ratchkovski, G. Anderson, G. A. Carver, A. J. Crone, T. E. Dawson, H. Fletcher, R. Hansen, E. L. Harp, R. A. Harris, D. P. Hill, S. Hreinsdottir, R. W. Jibson, L. M. Jones, R. Kayen, D. K. Keefer, C. F. Larsen, S. C. Moran, S. F. Personius, G. Plafker, B. Sherrod, K. Sieh, N. Sitar, and W. K. Wallace. The 2002 Denali fault earthquake, Alaska: a large magnitude, slip-partitioned event. Science, 300:1113-1118, 2003.Google ScholarCross Ref
- {6} E. Faccioli, F. Maggio, R. Paolucci, and A. Quarteroni. 2D and 3D elastic wave propagation by a pseudospectral domain decomposition method. J. Seismol., 1:237-251, 1997.Google ScholarCross Ref
- {7} C. Ji, D. V. Helmberger, and D. J. Wald. Preliminary slip history of the 2002 Denali earthquake. EOS Trans. AGU., 83, 2002.Google Scholar
- {8} D. Komatitsch. Méthodes spectrales et éléments spectraux pour l'équation de l'élastodynamique 2D et 3D en milieu hétérogène (Spectral and spectral-element methods for the 2D and 3D elastodynamics equations in heterogeneous media). PhD thesis, Institut de Physique du Globe, Paris, France, 1997.Google Scholar
- {9} D. Komatitsch and J. Tromp. Introduction to the spectral-element method for 3-D seismic wave propagation. Geophys. J. Int., 139:806-822, 1999.Google ScholarCross Ref
- {10} D. Komatitsch and J. Tromp. Spectral-element simulations of global seismic wave propagation-I. Validation. Geophys. J. Int., 149:390-412, 2002.Google ScholarCross Ref
- {11} D. Komatitsch and J. Tromp. Spectral-element simulations of global seismic wave propagation-II. 3-D models, oceans, rotation, and self-gravitation. Geophys. J. Int., 150:303-318, 2002.Google ScholarCross Ref
- {12} D. Komatitsch and J. P. Vilotte. The spectral-element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bull. Seismol. Soc. Am., 88(2):368-392, 1998.Google ScholarCross Ref
- {13} R. A. Page, G. Plafker, and H. Pulpan. Block rotation in east-central Alaska: a framework for evaluating earthquake potential? Geology, 23:629-632, 1995.Google ScholarCross Ref
- {14} J. Ritsema, H. J. Van Heijst, and J. H. Woodhouse. Complex shear velocity structure imaged beneath Africa and Iceland. Science, 286:1925-1928, 1999.Google ScholarCross Ref
- {15} C. Ronchi, R. Ianoco, and P. S. Paolucci. The "Cubed Sphere": a new method for the solution of partial differential equations in spherical geometry. J. Comput. Phys., 124:93-114, 1996. Google ScholarDigital Library
- {16} R. Sadourny. Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Monthly Weather Review, 100:136-144, 1972.Google ScholarCross Ref
- {17} G. Seriani. 3-D large-scale wave propagation modeling by a spectral element method on a Cray T3E multiprocessor. Comput. Methods Appl. Mech. Engrg., 164:235-247, 1998.Google ScholarCross Ref
- {18} S. Shingu, H. Takahara, H. Fuchigami, M. Yamada, Y. Tsuda, W. Ohfuchi, Y. Sasaki, K. Kobayashi, T. Hagiwara, S.-I. Habata, M. Yokokawa, H. Itoh, and K. Otsuka. A 26.58 teraflops global atmospheric simulation with the spectral transform method on the Earth Simulator. Proceedings of the ACM/IEEE Supercomputing SC'2002 conference, 2002. Published on CD-ROM and at www.sc-conference.org/sc2002. Google ScholarDigital Library
Recommendations
Software system of the earth simulator
The Earth Simulator (ES) is a large scale, distributed memory, parallel computer system consisting of 640 processor nodes (PN) with shared memory vector multiprocessors (64GFLOPS/PN, 5120 APs in total, AP: arithmetic processor). All the nodes are ...
Three-level hybrid vs. flat MPI on the Earth Simulator: Parallel iterative solvers for finite-element method
6th IMACS International symposium on iterative methods in scientific computingAn efficient parallel iterative method for finite element method has been developed for symmetric multiprocessor (SMP) cluster architectures with vector processors such as the Earth Simulator. The method is based on a three-level hybrid parallel ...
A 1.8 trillion degrees-of-freedom, 1.24 petaflops global seismic wave simulation on the K computer
We present high-performance simulations of global seismic wave propagation with an unprecedented accuracy of 1.2 s seismic period for a realistic three-dimensional Earth model using the spectral element method on the K computer. Our seismic simulations ...
Comments