Skip to main content

2018 | Buch

High Performance Computing in Science and Engineering ' 17

Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2017

herausgegeben von: Prof. Dr. Wolfgang E. Nagel, Prof. Dr. Dietmar H. Kröner, Prof. Dr. Michael M. Resch

Verlag: Springer International Publishing

insite
SUCHEN

Über dieses Buch

This book presents the state-of-the-art in supercomputer simulation. It includes the latest findings from leading researchers using systems from the High Performance Computing Center Stuttgart (HLRS) in 2017. The reports cover all fields of computational science and engineering ranging from CFD to computational physics and from chemistry to computer science with a special emphasis on industrially relevant applications. Presenting findings of one of Europe’s leading systems, this volume covers a wide variety of applications that deliver a high level of sustained performance.The book covers the main methods in high-performance computing. Its outstanding results in achieving the best performance for production codes are of particular interest for both scientists and engineers. The book comes with a wealth of color illustrations and tables of results.

Inhaltsverzeichnis

Frontmatter

Physics

Frontmatter
Blood Proteins and Their Interactions with Nanoparticles Investigated Using Molecular Dynamics Simulations

Blood proteins play a fundamental role in determining the response of the organism to the injection of drugs or, more in general, of therapeutic preparations in the blood stream. Some of these proteins are responsible for mediating immune response and coagulation. Nanoparticles, which are being intensely investigated as possible drug nanocarriers, heavily interact with blood proteins and their ultimate fate is determined by these interactions. Here we report the results of molecular dynamics simulations of several blood proteins aimed to determining their possible behavior at the nanoparticle surface. On one hand we investigated the behavior of fibrinogen, a glycoprotein, which polymerizes into fibrin during coagulation. On the other hand we investigated the behavior of several blood proteins in the presence of the polymer poly (ethylene-glycol), often used as nanoparticle coating to reduce unspecific interactions with the surrounding environment.

Timo Schäfer, Jiajia Zhou, Friederike Schmid, Giovanni Settanni
Cosmic Large-Scale Structure in the IllustrisTNG Simulations

We have finished two new, extremely large hydrodynamical simulations of galaxy formation that significantly advance the state of the art in cosmology. Together with accompanying dark matter only runs, we call them ‘IllustrisTNG’, the next generation Illustris simulations. Our largest and most ambitious calculation follows a cosmological volume 300 megaparsecs on a side and self-consistently solves the equations of magnetohydrodynamics and self-gravity coupled to the fundamental physical processes driving galaxy formation. We have employed AREPO, a sophisticated moving-mesh code developed by our team over the past 7 years and equipped with an improved, multi-purpose galaxy formation physics model. The simulated universe contains tens of thousands of galaxies encompassing a variety of environments, mass scales and evolutionary stages. The groundbreaking volume of TNG enables us to sample statistically significant sets of rare astrophysical objects like rich galaxy clusters, and to study galaxy formation and the spatial clustering of matter over a very large range of spatial scales. Here we report some early results on the matter and galaxy clustering found in the simulations. The two-point galaxy correlation function of our largest simulation agrees extremely well with the best available observational constraints from the Sloan Digital Sky Survey, both as a function of galaxy stellar mass and color. The predicted impact of baryonic physics on the matter power spectrum is sizeable and needs to be taken into account in precision studies of cosmology. Interestingly, this impact appears to be fairly robust to the details of the modelling of supermassive black holes, provided this reproduces the scaling properties of the intracluster medium of galaxy clusters.

Volker Springel, Annalisa Pillepich, Rainer Weinberger, Rüdiger Pakmor, Lars Hernquist, Dylan Nelson, Shy Genel, Mark Vogelsberger, Federico Marinacci, Jill Naiman, Paul Torrey
PAMOP: Large-Scale Calculations Supporting Experiments and Astrophysical Applications

Our prime computation effort is to support current and future measurements of atomic photoionization cross-sections being performed at various synchrotron radiation facilities around the globe, and computations for astrophysical applications. In our work we solve the Schrödinger or Dirac equation using the R-matrix or R-matrix with pseudo-states approach from first principles. The time dependent close-coupling (TDCC) method is also used in our work. Finally, we present cross-sections and rates determined for diatom-diatom and radiative collision processes between atoms and ions currently of great interest to astrophysics.

B. M. McLaughlin, C. P. Ballance, M. S. Pindzola, P. C. Stancil, J. F. Babb, S. Schippers, A. Müller
Phenomenology of Strange Resonances
Constraining the Hadronic Spectrum Through QCD Thermodynamics on the Lattice

Fluctuations of conserved charges allow to study the chemical composition of hadronic matter. A comparison between lattice simulations and the Hadron Resonance Gas (HRG) model suggested the existence of missing strange resonances. To clarify this issue we calculate the partial pressures of mesons and baryons with different strangeness quantum numbers using lattice simulations in the confined phase of QCD. In order to make this calculation feasible, we perform simulations at imaginary strangeness chemical potentials. We systematically study the effect of different hadronic spectra on thermodynamic observables in the HRG model and compare to lattice QCD results. We show that, for each hadronic sector, the well established states are not enough in order to have agreement with the lattice results. Additional states, either listed in the Particle Data Group booklet (PDG) but not well established, or predicted by the Quark Model (QM), are necessary in order to reproduce the lattice data. For mesons, it appears that the PDG and the quark model do not list enough strange mesons, or that, in this sector, interactions beyond those included in the HRG model are needed to reproduce the lattice QCD results.

Paolo Alba, Rene Bellwied, Szabolcs Borsanyi, Zoltan Fodor, Jana Günther, Sandor D. Katz, Valentina Mantovani Sarti, Jacquelyn Noronha-Hostler, Paolo Parotto, Attila Pasztor, Israel Portillo Vazquez, Claudia Ratti
Real-Time Lattice Simulations of Quantum Anomalies, Topologies and Particle Production in Strongly Correlated Gauge Theories

We present results of numerical lattice simulations of anomalous and topological effects in Quantum Electrodynamics (QED) and Quantum Chromodynamics (QCD) in far from equilibrium situations. Based on the classical-statistical approximation to the Schwinger-Keldysh path integral formalism, we perform extensive numerical studies including dynamical Wilson and overlap fermions. Using advanced algorithmic techniques, we study the real-time dynamics of the axial anomaly relevant for strong field laser physics beyond the Schwinger limit and we observe novel dynamical refringence effects caused by the anomaly. Furthermore, motivated by recent interest in the physics of the Chiral Magnetic Effect in ultra-relativistic heavy ion collisions, we study the real time dynamics of fermions during and after a sphaleron transition and anomalous transport in the presence of strong magnetic fields.

Niklas Mueller, Oscar Garcia-Montero, Naoto Tanji, Juergen Berges
Many-Body Effects in Fragmented, Depleted, and Condensed Bosonic Systems in Traps and Optical Cavities by MCTDHB and MCTDH-X

The many-body physics of trapped Bose-Einstein condensates (BECs) is very rich and demanding. During the past year of the MCTDHB project at the HLRS we continued to shed further light on it with the help of the MultiConfigurational Time-Dependent Hartree for Bosons (MCTDHB) method and using the MCTDHB and MCTDH-X software packages. Indeed, our results on which we report below span a realm of many-body effects in fragmented, depleted, and even in fully condensed BECs. Our findings include: (1) fragmented superradiance of a BEC trapped in an optical cavity; (2) properties of phantom (fragmented) vortices in trapped BECs; (3) dynamics of a two-dimensional trapped BEC described by the Bose-Hubbard Hamiltonian with MCTDH-X; (4) overlap of exact and Gross-Pitaevskii wave-functions in trapped BECs; (5) properties of the uncertainty product of an out-of-equilibrium trapped BEC; (6) many-body excitations and de-excitations in trapped BECs and relation to variance; and (7) many-body effects in the excitation spectrum of weakly-interacting BECs in finite one-dimensional optical lattices. These are all appealing and fundamental many-body results made through the kind allocation of computer resources by the HLRS to the MCTDHB project. Finally, we put forward some future developments and research plans, as well as further many-body perspectives.

Ofir E. Alon, Raphael Beinke, Christoph Bruder, Lorenz S. Cederbaum, Shachar Klaiman, Axel U. J. Lode, Kaspar Sakmann, Marcus Theisen, Marios C. Tsatsos, Storm E. Weiner, Alexej I. Streltsov

Molecules, Interfaces, and Solids

Frontmatter
Calculation of Global, High-Dimensional Potential Energy Surface Fits in Sum-of-Products Form Using Monte-Carlo Methods

We have implemented a Monte-Carlo version of the well-known potfit algorithm. With potfit one can transform high-dimensional potential energy surfaces sampled on a grid into a sum-of-products form. More precisely, an fth order general tensor can be transformed into Tucker form. Using Monte-Carlo methods we avoid high-dimensional integrals that are needed to obtain optimal fits and simultaneously introduce importance sampling. The Tucker form is well suited for further use within the Heidelberg MCTDH package for solving the time-dependent as well as the time-independent Schrödinger equation of molecular systems. We demonstrate the power of the Monte-Carlo potfit algorithm by globally fitting the 15-dimensional potential energy surface of the Zundel cation (H5O2+ $$_2^+$$ ) and subsequently calculating the lowest vibrational eigenstates of the molecule.

Markus Schröder, Hans-Dieter Meyer
Sum Frequency Generation Spectra from Velocity-Velocity Correlation Functions: New Developments and Applications

At the interface, the properties of water can be rather different from those observed in the bulk. In this chapter we present an overview of our computational approach to understand water structure and dynamics at the interface including atomistic and electronic structure details. In particular we show how Density Functional Theory-based molecular dynamics simulations (DFT-MD) of water interfaces can provide a microscopic interpretation of recent experimental results from surface sensitive vibrational Sum Frequency Generation spectroscopy (SFG). In our recent work we developed an expression for the calculation of the SFG spectra of water interfaces which is based on the projection of the atomic velocities on the local normal modes. Our approach permits to obtain the SFG signal from suitable velocity-velocity correlation functions, reducing the computational cost to that of the accumulation of a molecular dynamics trajectory, and therefore cutting the overhead costs associated to the explicit calculation of the dipole moment and polarizability tensor. Our method permits to interpret the peaks in the spectrum in terms of local modes, also including the bending region. The results for the water-air interface, obtained using extensive ab initio molecular dynamics simulations over 400 ns, are discussed in connection to recent phase resolved experimental data.

Khatib Rémi, Sulpizi Marialore
Photo-Excited Surface Dynamics from Massively Parallel Constrained-DFT Calculations

Constrained density-functional theory (DFT) calculations show that the recently observed optically induced insulator-metal transition of the In/Si(111)(8×2)/(4×1) nanowire array (Frigge et al., Nature 544:207, 2017) corresponds to the non-thermal melting of a charge-density wave (CDW). Massively parallel numerical simulations allow for the simulation of the photo-excited nanowires and provide a detailed microscopic understanding of the CDW melting process in terms of electronic surface bands and selectively excited soft phonon modes. Excited-state molecular dynamics in adiabatic approximation shows that the insulator-metal transition can be as fast as 350 fs.

A. Lücke, T. Biktagirov, A. Riefer, M. Landmann, M. Rohrmüller, C. Braun, S. Neufeld, U. Gerstmann, W. G. Schmidt
Dynamic Material Parameters in Molecular Dynamics and Hydrodynamic Simulations on Ultrashort-Pulse Laser Ablation of Aluminum

Molecular dynamics reveals a detailed insight into the material processes. Among various available codes, IMD features an implementation of the two-temperature model for laser-matter interaction. Reliable simulations, however, are restricted to the femtosecond regime, since a constant absorptivity is assumed. For picosecond pulses, changes of the dielectric permittivity ?? and the electron thermal conductivity κ e due to temperature, density and mean charge have to be considered. Therefore, IMD algorithms were modified for the dynamic recalculation of ?? and κ e for every timestep following the corresponding implementation in the hydrodynamic code Polly-2T. The usage of dynamic permittivity yields an enhanced absorptivity during the pulse leading to greater material heating. In contrast, increasing conductivity induces material cooling which in turn decreases absorptivity and heating resulting in a higher ablation threshold. This underlines the importance of a dynamic model for ?? and κ e with longer pulses which is commonly often neglected. Summarizing all simulations with respect to absorbed laser fluence, ablation depths in Polly-2T are two times higher than in IMD. This can be ascribed to the higher spallation strength in IMD stemming from the material-specific potential deviating from the equations of state used in Polly-2T.

Stefan Scharring, Marco Patrizio, Hans-Albert Eckel, Johannes Roth, Mikhail Povarnitsyn

Reactive Flows

Frontmatter
Automated Code Generation for Maximizing Performance of Detailed Chemistry Calculations in OpenFOAM

In direct numerical simulation of turbulent combustion, the majority of the total simulation time is often spent on evaluating chemical reaction rates from detailed reaction mechanisms. In this work, an optimization method is presented for speeding up the calculation of chemical reaction rates significantly, which has been implemented into the open-source CFD code OpenFOAM. A converter tool has been developed, which translates any input file containing chemical reaction mechanisms into C++ source code. The automatically generated code allows to restructure the reaction mechanisms for efficient computation and enables more compiler optimizations. Additional performance improvements are achieved by generating densely packed data and linear access patterns that can be vectorized in order to exploit the maximum performance on HPC systems. The generated source code compiles to an OpenFOAM library, which can directly be used in simulations through OpenFOAM’s runtime selection mechanism. The optimization concept has been applied to a realistic combustion case simulated on two peta-scale supercomputers, among them the fastest HPC cluster Hazel Hen (Cray XC40) in Germany. The optimized code leads to a decrease of total simulation time of up to 40% and this improvement increases with the complexity of the involved chemical reactions. Moreover, the optimized code yields good parallel performance on up to 28,800 CPU cores.

Thorsten Zirwes, Feichi Zhang, Jordan A. Denev, Peter Habisreuther, Henning Bockhorn
A Resolved Simulation Study on the Interactions Between Droplets and Turbulent Flames Using OpenFOAM

This study presents direct numerical simulations (DNS) of turbulent reacting flows around evaporating single fuel droplets and droplet arrays. Statistical analysis of interactions between the droplets and the turbulent flames are used to develop sub-grid scale models for mixture fraction based approaches such as flamelet or conditional moment closure (CMC) methods. The specific challenges are posed by the effects of the evaporating spray on the composition field in inter-droplet space and by the presence of combustion. Here, we analyse the best possible setup for such a fully resolved DNS. The numerical constraints are given by (1) the need to resolve all small scale effects, i.e. the smallest turbulent eddies and the boundary layer thickness around the droplets, and (2) the desire to include scales covering the entire turbulence spectrum to ensure a realistic interaction between the large and small scales. The largest scales are typically limited by the size of the computational domain, and these two demands (high resolution and large domain size) can easily lead to extensive computational requirements. We suggest an optimal setup for fully resolved DNS that ensures a good balance between computational cost and solution accuracy. The optimal mesh resolution and domain size do not introduce any bias for the analysis of characteristic quantities such as mixture fraction, its PDF and conditional scalar dissipation. Further, adequate scalability of OpenFOAM for the different setups is reported.

B. Wang, H. Chu, A. Kronenburg, O. T. Stein
Towards Affordable LES of Rocket Combustion Engines

Due to the large disparity of length and time scales in rocket combustors corresponding LES are highly time consuming. Steps towards simulating rocket combustion engines with affordable LES with the compressible, implicit combustion code TASCOM3D are presented. DES-family (DES, DDES, iDDES) models are compared and evaluated on two simple test cases: the turbulent isotropic homogeneous turbulence and the planer channel flow. iDDES shows promising results to be used in future rocket combustion engine simulations and can make LES affordable in these kind of combustion simulations.

Roman Keller, Martin Grader, Peter Gerlinger, Manfred Aigner

Computational Fluid Dynamics

Frontmatter
Numerical Analysis of Heat Transfer During Cooling of Supercritical Fluid by Means of Direct Numerical Simulation

Supercritical fluids have a wide spectrum of application, ranging from power generation to enhanced oil extraction. The sensitive nature of thermophysical properties makes the heat transfer complicated. Therefore, in this work, an investigation is made for the vertically-oriented pipe to understand the physics behind the heat transfer deterioration occurring during cooling. For that, carbon dioxide is chosen as working fluid, and direct numerical simulations with the open source finite volume code OpenFOAM have been performed with variation in the strength of body force due to buoyancy. It was found out that body force affects the axial temperature profile. Initial examination of turbulence statistics revealed that turbulence is modulated by buoyancy and deacceleration. Further investigation unveiled that long 1-dimensional structures characterized by streak elongation were present in the downward flow. In the end, Octant analysis indicates the reduction in ejection and sweep events for downward flow caused the decrease in turbulence.

Sandeep Pandey, Xu Chu, Eckart Laurien
Control of Traveling Crossflow Vortices Using Plasma Actuators

It has been shown recently by direct numerical simulations that plasma actuators can be used to delay laminar-turbulent transition caused by steady crossflow vortices (CFVs) in three-dimensional boundary layers on swept aerodynamic surfaces. In the current work the applicability of such actuators to control transition caused by traveling CFVs is explored by two techniques. In the first technique, named upstream flow deformation, the actuators are used to excite steady CFV control modes. The resulting narrow spaced control CFVs induce a beneficial mean-flow distortion and weaken the primary crossflow instability, yielding delayed transition. In the second technique, the direct attenuation of nonlinear traveling CFVs, the actuators are positioned more downstream, where the traveling CFVs have already established. The localized unsteady forcing against the direction of the crossflow is then aimed at attenuating the amplitude of the traveling CFVs by directly tackling the three-dimensional nonlinear disturbance state. With both techniques transition can be delayed, however with a significantly higher efficiency for the method of upstream flow deformation.

Philipp C. Dörr, Zhengfei Guo, Johannes M. F. Peter, Markus J. Kloker
Towards the Implementation of a New Multigrid Solver in the DNS Code FS3D for Simulations of Shear-Thinning Jet Break-Up at Higher Reynolds Numbers

Liquid jet break-up appears in many technical applications, as well as in nature. It consists of complex physical processes, which happen on very small scales in space and time. This makes them hard to capture by experimental methods; and therefore a prime subject for numerical investigations. The state-of-the-art approach combines the Volume of Fluid (VOF) method with Direct Numerical Simulations (DNS) as employed in the ITLR in-house code Free Surface 3D (FS3D). The simulation of these jets is dependent on very fine grids, with most of the computational costs incurred by solving the Pressure Poisson Equation. In order to simulate larger computational domains, we tried to improve the performance of FS3D by the implementation of a new multigrid solver. For this we selected the solver contained in the UG4 package developed by the Goethe Center for Scientific Computing at the University of Frankfurt. We will show simulations of the primary break-up of shear-thinning liquid jets and explain why larger computational domains are necessary. Results are preliminary. We demonstrate that the implementation of UG4 into FS3D provides a noticeable increase in weak scaling performance, while the change in strong scaling is yet detrimental. We will then discuss ways to further improve these results.

Moritz Ertl, Jonathan Reutzsch, Arne Nägel, Gabriel Wittum, Bernhard Weigand
Numerical Investigation of the Turbulent Wake of Generic Space Launchers

The turbulent wakes of generic space launchers are numerically investigated via a zonal RANS/LES method and optimized dynamic mode decomposition (DMD), to gain insight into characteristic wake flow modes being responsible for asymmetrical loads on the engine extension known as buffet loads. The considered launcher geometries range from planar space launchers up to axisymmetric free flight configurations investigated at varying free stream conditions, i.e. transonic and supersonic. The investigated wake topologies reveal a highly unsteady behavior of the shear layer and the separation region resulting in strongly periodic and antisymmetric wall pressure fluctuations on the nozzle surface. Using conventional spectral analysis and dynamic mode decomposition, several spatio-temporal coherent low frequency modes which are responsible for the detected pressure oscillations are identified. In addition, a passive flow control device consisting of semi-circular lobes integrated at the base shoulder of the planar configuration is investigated. The objective of the concept is to reduce the reattachment length and thus the lever arm of the forces as well as to stabilize the separated shear layer. The results show a significant reduction of the reattachment length by about 75%. In addition, the semi-circular lobes partially reduce undesired low frequency pressure fluctuations on the nozzle surface. However, this reduction is achieved at the expense of an increase of high frequency pressure fluctuations due to intensified small turbulent scales.

S. Loosen, V. Statnikov, M. Meinke, W. Schröder
Optimization and HPC-Applications of the Flow Solver FLOWer

Recent optimizations and HPC-applications of the flow solver FLOWer are presented in this paper. A graph partitioning method is introduced to the MPI communication, which reduces the number of messages as well as the total message size, leading to a run time speed-up of 20%. A numerical investigation of a finite wing shows the influence of the wind tunnel wall only in the wing root area and agrees well with experimental data for attached flow. Both a URANS and a Delayed Detached-Eddy Simulation (DDES) of the massively stalled wing reveal difficulties in matching the experimental behaviour of flow separation. Finally, a simulation of a model Contra-Rotating Open Rotor (CROR) at various operating conditions exhibit interaction effects, blade loadings and noise emissions which agree well with expectations and results from literature.

Johannes Letzgus, Lukas Dürrwächter, Ulrich Schäferlein, Manuel Keßler, Ewald Krämer
Numerical Simulations of Artificial Disturbance Influence on a High Lift Airfoil

Numerical simulation results of the DLR-F15 high lift airfoil with statically and dynamically disturbed inflow conditions are presented and analysed. The two-dimensionally extruded high-lift airfoil, consisting of main element and flap, is exposed to the influence of a static tip vortex in order to generate a three-dimensional inflow condition. In a second step the static distortion is superimposed by an additional, artificial and dynamic gust. The set-up is a wind tunnel test section, where a pitching airfoil, which serves as a gust generator, and a finite wing, which serves as a vortex generator, are located upstream of the high lift airfoil. Unsteady RANS simulations with the inflow Mach number M = 0.15 and Reynolds number Re = 2.0 million were performed. The high lift airfoil angle of attack is α = 0∘. The results give an insight into the different emerging effects and the reaction of the high lift airfoil.

Katharina Wawrzinek, Thorsten Lutz, Ewald Krämer
About the Influence of Wind Tunnel Walls, Tower and Nozzle on the Performance of a Model Wind Turbine

The influence of wind tunnel walls, tower and nozzle on the performance of a model wind turbine is investigated in the present paper using Computational Fluid Dynamics (CFD). The model wind turbine has a radius of 1.5 m and is located in a wind tunnel with a cross section of 4.2 m × 4.2 m. Global loads, angle of attack distributions as well as flow fields are compared to each other to evaluate the influence of the different configurations.

Annette Klein (née Fischer), Sven Zabel, Thorsten Lutz, Ewald Krämer
Numerical Analysis of a Propeller Turbine Operated in Part Load Conditions

A part load operating point of a hydraulic propeller turbine is numerically analyzed. Transient flow simulations without geometry simplifications are performed on meshes up to 100 million nodes using RANS and hybrid RANS-LES turbulence models. The weaknesses of steady state and transient RANS computations are evaluated against transient approaches with hybrid RANS-LES turbulence models. Due to the dominating phenomena of the vortex rope the focus of the results presented is on the draft tube flow field. Therefore, velocity profiles at 5 evaluation lines located in the draft tube are compared as well as the capabilities of the turbulence models to resolve the turbulent flow structures and developing vorticies for different mesh densities. Additionally, hydraulic losses in the machine components are compared.

Bernd Junginger, Stefan Riedelbauch
Mesoscale Simulations of Janus Particles and Deformable Capsules in Flow

Complex fluids are common in our daily life and play an important role in many industrial applications. The understanding of the dynamical properties of these fluids and interfacial effects is still lacking. Computer simulations pose an attractive way to gain insight into the underlying physics. In this report we restrict ourselves to two examples of complex fluids and their simulation by means of numerical schemes coupled to the lattice Boltzmann method as a solver for the hydrodynamics of the problem. First, we study Janus particles at a fluid-fluid interface using the Shan-Chen pseudopotential approach for multicomponent fluids in combination with a discrete element algorithm. Second, we study the dense suspension of deformable capsules in a Kolmogorov flow by combining the lattice Boltzmann method with the immersed boundary method.

Othmane Aouane, Qingguang Xie, Andrea Scagliarini, Jens Harting
Application and Development of the High Order Discontinuous Galerkin Spectral Element Method for Compressible Multiscale Flows

This paper summarizes our progress in the application of a high-order discontinuous Galerkin (DG) method for scale resolving fluid dynamics simulations on the Cray XC40 Hazel Hen cluster at HLRS. We present the large eddy simulation (LES) of flow around a wall mounted cylinder, a LES of flow around an airfoil at realistic Reynolds number using a recently introduced kinetic energy preserving flux formulation and a simulation of transitional flow in a low pressure turbine. Furthermore, it provides an overview over the parallel efficiency reached by our code when using up to 49, 152 CPUs and the latest developments of our DG framework.

Andrea Beck, Thomas Bolemann, David Flad, Hannes Frank, Nico Krais, Kristina Kukuschkin, Matthias Sonntag, Claus-Dieter Munz

Transport and Climate

Frontmatter
Regional Climate Simulations with COSMO-CLM: Ensembles, Very High Resolution and Paleoclimate

The IMK-TRO (KIT) presents in the HLRS annual report for 2016–2017 projects and their results using the CRAY XC40 “Hazel Hen”. The research focuses on the very high resolution regional climate simulations including the modeling of land surface processes and urban climate, the generation of ensemble projections, and regional paleoclimate (PALMOD). The simulations are performed with the regional climate model (RCM) COSMO-CLM (CCLM) and cover spatial resolutions from 50 to 2.8 km. Within the projects, the standard CCLM is enhanced; for the analysis of the impact of different soil-vegetation transfer schemes (SVATs) VEG3D is coupled via OASIS3-MCT to CCLM. For the PALMOD project, a special isotope-enabled version CCLMiso is used. To highlight the added value, the results of the higher resolution climate predictions are compared to those of simulations with coarser resolutions. In addition, the impact of different global driving data sets is investigated. Climate projections are performed for two future time slices, 2021–2050 and 2071–2100. The urban climate and its change are also investigated using very high resolution simulations to enable the energetic optimisation of buildings. The required Wall-Clock-Times (WCT) range from 9 to 2000 node-hours per simulated year.

G. Schädler, H.-J. Panitz, E. Christner, H. Feldmann, M. Karremann, N. Laube
High Resolution WRF Simulations for Climate Change Studies in Germany

The scope of WRFCLIM is to produce high resolution regional climate simulations with WRF from 1958 to 2100 in the framework of the BMBF funded ReKliEs-De (Regionales Klimaensemble für Deutschland) and the DFG funded research unit on regional climate change (FOR 1695) on a 0.11∘ (approx. 12 km) grid and further downscale one projection to 0.0275∘ (approx. 3 km) resolution within the FOR 1695 project from 2000 to 2040.The overall goal of ReKliEs-De is to derive robust climate change information on high spatial resolution for Germany and the large river catchments draining into Germany from this unique model ensemble. The University of Hohenheim (UHOH) contributes five simulations using WRF to the joint Global Climate Model-Regional Climate Model-matrix, including pre- and post-processing of model input and output data, evaluation of the single model results and delivery of the resulting data to the partners for further post-processing, distribution and storage. The scientific focus of the analyses performed at UHOH is the estimation of the bandwidth of the ensemble results in close cooperation with the project partners.It is the joint objective of the Research Unit on regional climate change to investigate the effects of global climate change on structure and functions of agricultural landscapes on a regional scale and to work out projections for their development until 2040 taking into consideration various possible socio-economic conditions and adaptation processes. In order to achieve these objectives, the WRF model is coupled with land surface and crop models as well as with multi agent systems in a new integrated land-system model system, and structure and parameterisation of various model components are optimised. Within WRFCLIM verification runs of WRF with and without the crop model GECROS, which includes recent improvements with respect to the representation of agricultural land cover, over Central Europe with convection-permitting (CP) resolution will be simulated downscaling ERA-Interim forced WRFCLIM simulation data from 0.11∘ are currently ongoing.

Kirsten Warrach-Sagi, Viktoria Mohr, Volker Wulfmeyer
Seasonal Simulation of Weather Extremes

To date, seasonal forecasts are often performed by applying horizontal resolutions of 75–150 km due to lack of computational resources and associated operational constraints. As this resolution is too coarse to represent fine scale structures impacting the large scale circulation, a convection permitting (CP) resolution of less than 4 km horizontal resolution is required. Most of the simulations are carried out as limited area model (LAMs) and thus require boundary conditions at all four domain boundaries. In this study, the Weather Research and Forecasting (WRF) model is applied in a latitude belt set-up in order to avoid the application of zonal boundaries. The horizontal resolution is 0.03∘ spanning a belt between 65∘ N and 57∘ S encompassing 12000 ∗ 4060 ∗ 57 grid boxes. The simulation is forced by ECMWF analysis data and high resolution SST data from multiple sources. The simulation period is February to beginning of July 2015 which is a strong El Niño period.

Thomas Schwitalla, Volker Wulfmeyer, Kirsten Warrach-Sagi

Miscellaneous Topics

Frontmatter
Massively Parallel Multigrid for the Simulation of Skin Permeation on Anisotropic Tetrakaidecahedral Cell Geometries

Numerical simulation based on mathematical models is an important pillar for enhancing the understanding of permeation processes in the skin. To adequately resolve the complex geometrical structure of the skin, special models based on tetrakaidecahedral cells have been suggested. While these models preserve many of the desirable properties of the underlying geometry, they impose challenges regarding mesh generation and solver robustness.To improve robustness of the used multigrid solver, we propose a new mesh and hierarchy structure with good aspect ratios and angle conditions. Furthermore, we show how those meshes can be used in scalable massively parallel multigrid based computations of permeation processes in the skin.

Sebastian Reiter, Arne Nägel, Andreas Vogel, Gabriel Wittum
Real Data Applications of Seismic Full Waveform Inversion

Full waveform inversion (FWI) is a powerful imaging technique which exploits the richness of seismic waveforms. We further developed FWI to obtain multi-parameter images at high resolution. Here, we involve physical parameters, such as velocities and attenuation of seismic waves as well as mass density, which are essential for a reliable petrophysical characterization of subsurface structures in hydrocarbon exploration, geotechnical applications and underground constructions. Referring to this, we successfully applied FWI to field datasets recorded in the Black Sea and in the shallow-water area of a river delta in the Atlantic Ocean. We obtained detailed subsurface images containing rock formations which might be potential gas deposits. Additionally, we performed synthetic studies as preparatory steps to verify methodological improvements for further field-data applications. Here, we demonstrate resolution capabilities of FWI for imaging geological structures beneath salt bodies, investigate strategies to recover attenuation information from seismic data and perform a joint inversion of surface waves to image the very shallow subsurface.

A. Kurzmann, L. Gaßner, R. Shigapov, N. Thiel, N. Athanasopoulos, T. Bohlen, T. Steinweg
The Impact of Pores on Microstructure Evolution: A Phase-Field Study of Pore-Grain Boundary Interaction

Among the most important issues of today’s materials research ceramic materials play a key role as e.g. in Lithium batteries, in fuel cells or in photovoltaics. For all these applications a tailored microstructure is needed, which usually requires sintering: A pressed body of compacted powder redistributes its material and shrinks to a compact body without pores. In a very porous polycrystal, pores constrain the motion of interfaces (pore drag) and no grain growth occurs. During further sintering the number and size of pores decreases and the pore drag effect fades away. Accordingly, in the final stage of sintering grain growth emerges. This grain growth decreases the driving force for sintering and is undesirable, but hard to avoid. Since application of ceramic materials usually requires a dense and fine-grained microstructure, it is of high interest to control the interplay of remaining pores and interface migration during sintering.Unfortunately, the present modeling of sintering does not allow for predicting microstructural evolution in an adequate way. In this study, a previously developed phase-field model of pore-grain boundary interaction during final stage sintering is extended by a pore-interaction module to improve the modeling of final stage sintering. This module handles the growth of pores that come into contact during grain growth.The model with the extensions is used to simulate interface migration in a well-defined model setup. The results show that the present model is appropriate to describe grain growth during the final stage of sintering. However, the need of large scale simulations becomes evident: pore drag depends critically on the local geometry (i.e. position of the pores at grain boundaries, triple lines or quadruple points). The microstructure evolution during final stage sintering in polycrystalline ceramics underlies strong statistical variations in the local geometry. Accordingly, if grain growth in polycrystals in the presence of remaining pores from sintering is considered in detail, large scale simulations are needed to picture the local statistic variation of pore drag in an adequate way.

V. Rehn, J. Hötzer, M. Kellner, M. Seiz, C. Serr, W. Rheinheimer, M. J. Hoffmann, B. Nestler
Modeling the Interior Dynamics of Terrestrial Planets

Over the past years, large scale numerical simulations of planetary interiors have become an important tool to understand physical processes responsible for the surface features observed by various space missions visiting the terrestrial planets of our Solar System. Such large scale applications need to show good scalability on thousands of computational cores while handling a considerable amount of data that needs to be read from and stored to a file system. To this end, we analyzed numerous approaches to write files on the Cray XC40 Hazel Hen supercomputer. Our study shows that HPC applications parallelized using MPI highly benefit from utilizing the MPI I/O facilities. By implementing MPI I/O in Gaia, we improved the I/O performance up to a factor of 100. Additionally, in this study we present applications of the fluid flow solver Gaia using high resolution regional spherical shell grids to study the interior dynamics and thermal evolution of terrestrial bodies of our Solar System.

Ana-Catalina Plesa, Christian Hüttig, Florian Willich
EXAHD: An Exa-Scalable Two-Level Sparse Grid Approach for Higher-Dimensional Problems in Plasma Physics and Beyond

Within the current reporting period (04/2016–04/2017) of our HLRS project we have developed a scalable implementation of the fault-tolerant combination technique. Fault-tolerance is one of the key topics in the ongoing research of algorithms for future exascale systems. Our algorithms enable fault-tolerance for both hard and soft faults, for the efficient and massively parallel computation of high-dimensional PDEs without the need of checkpointing or process replication. The research project EXAHD is part of DFG’s priority program “Software for Exascale Computing” (SPPEXA). The project’s target application is the large-scale simulation of plasma turbulence with the code GENE. The report combines parts of three publications.

Mario Heene, Alfredo Parra Hinojosa, Michael Obersteiner, Hans-Joachim Bungartz, Dirk Pflüger
Metadaten
Titel
High Performance Computing in Science and Engineering ' 17
herausgegeben von
Prof. Dr. Wolfgang E. Nagel
Prof. Dr. Dietmar H. Kröner
Prof. Dr. Michael M. Resch
Copyright-Jahr
2018
Electronic ISBN
978-3-319-68394-2
Print ISBN
978-3-319-68393-5
DOI
https://doi.org/10.1007/978-3-319-68394-2