Skip to main content

About this book

This book presents the state-of-the-art in supercomputer simulation. It includes the latest findings from leading researchers using systems from the High Performance Computing Center Stuttgart (HLRS) in 2018. The reports cover all fields of computational science and engineering ranging from CFD to computational physics and from chemistry to computer science with a special emphasis on industrially relevant applications. Presenting findings of one of Europe’s leading systems, this volume covers a wide variety of applications that deliver a high level of sustained performance.
The book covers the main methods in high-performance computing. Its outstanding results in achieving the best performance for production codes are of particular interest for both scientists and engineers. The book comes with a wealth of color illustrations and tables of results.

Table of Contents




The TNG50 Simulation of the IllustrisTNG Project: Bridging the Gap Between Large Cosmological Volumes and Resolved Galaxies

Cosmological hydrodynamical simulations of galaxy formation are a powerful theoretical tool, and enable us to directly calculate the observable signatures resulting from the complex process of cosmic structure formation. Here we present early results from the ongoing TNG50 run, an unprecedented ‘next generation’ cosmological, magnetohydrodynamical simulation—the third and final volume of the IllustrisTNG project, with over 20 billion resolution elements and capturing spatial scales of $$\sim $$ 100 parsecs. It incorporates the comprehensive TNG model for galaxy formation physics, and we here describe the simulation scope, novel achievements, and early investigations on resolved galactic and halo structural properties.

Dylan Nelson, Annalisa Pillepich, Volker Springel, Rüdiger Pakmor, Lars Hernquist, Rainer Weinberger, Shy Genel, Mark Vogelsberger, Federico Marinacci, Paul Torrey, Jill Naiman

44065 HypeBBH Yearly Report—High Performance Computing Services at HLRS

The allocation “44065 HypeBBHs” on the system CRAY XC40 (HAZEL HEN) at HLRS has been awarded to our research group in February 2015 for one year, then extended in February 2016 until December 2016, becoming one of the main computing resources available to us. A request for a further extension until December 2017 and new resources allocation has been submitted, and we have been granted access to the machine with full approval of our application.

Federico Guercilena, Elias Most, Luciano Rezzolla

PAMOP2: Towards Exascale Computations Supporting Experiments and Astrophysics

Our prime computational effort is to support current and future measurements of atomic photoionization cross-sections at various synchrotron radiation facilities, and ion-atom collision experiments, together with plasma, fusion and astrophysical applications. In our work we solve the Schrödinger or Dirac equation using the R-matrix or R-matrix with pseudo-states approach from first principles. Finally, we present cross-sections and rates for radiative charge transfer, radiative association, and photodissociation collision processes between atoms and ions of interest for several astrophysical applications.

B. M. McLaughlin, C. P. Ballance, M. S. Pindzola, P. C. Stancil, J. F. Babb, S. Schippers, A. Müller

Interactions Between Blood Proteins and Nanoparticles Investigated Using Molecular Dynamics Simulations

In the development of new therapeutic agents based on nanoparticles it is of fundamental importance understanding how these substances interact with the underlying biological milieu. Our research is focussed on simulating in silico these interactions using accurate atomistic models, and gather from these information general pictures and simplified models of the underlying phenomena. Here we report results about the interactions of blood proteins with promising hydrophilic polymers used for the coating of therapeutic nanoparticles, about the salt dependent behavior of one of these polymers (poly-(ethylene glycol)) and about the interactions of blood proteins with silica, one of the most used materials for the production of nanoparticles.

Timo Schafer, Christian Muhl, Matthias Barz, Friederike Schmid, Giovanni Settanni

The QCD Phase Diagram from the Lattice

Higher Order Fluctuations and Correlations of Conserved Charges from Lattice QCD Status Report: POSR 44090

We calculate several diagonal and non-diagonal fluctuations of conserved charges in a system of $$2+1+1$$ quark flavors with physical masses, on a lattice with size $$48^3\times 12$$ . Higher order fluctuations at $$\mu _B=0$$ are obtained as derivatives of the lower order ones, simulated at imaginary chemical potential. From these correlations and fluctuations we construct ratios of net-baryon number cumulants as functions of temperature and chemical potential, which satisfy the experimental conditions of strangeness neutrality and proton/baryon ratio. Our results qualitatively explain the behavior of the measured cumulant ratios by the STAR collaboration. We explain the obtained simulation results with a simple model, and find consistent behaviour with a scenario with no nearby critical end point in the QCD phase diagram.

Szabolcs Borsanyi, Zoltan Fodor, Jana Günther, Sandor D. Katz, Attila Pasztor, Israel P. Vazquez, Claudia Ratti, K. K. Szabó

Exploring Many-Body Physics with Bose-Einstein Condensates

Bose-Einstein condensates (BECs) offer a fruitful, often uncharted ground for exploring physics of many-particle systems. In the present year of the MCTDHB project at the HLRS, we maintained and extended our investigations of BECs and interacting bosonic systems using the MultiConfigurational Time-Dependent Hartree for Bosons (MCTDHB) method and running the MCTDHB and MCTDH-X software packages on the Cray XC40 system Hazel Hen. The results we disseminate within this report comprise: (i) Entropies and correlations of ultracold bosons in a lattice; (ii) Crystallization of bosons with dipole-dipole interactions and its detection in single-shot images; (iii) Management of correlations in ultracold gases; (iv) Pulverizing a BEC; (v) Dynamical pulsation of ultracold droplet crystals by laser light; (vi) Two-component bosons interacting with photons in a cavity; (vii) Quantum dynamics of a bosonic Josephson junction and the impact of the range of the interaction; (viii) Trapped bosons in the infinite-particle limit and their exact many-body wavefunction and properties; (ix) Angular-momentum conservation in a BEC and Gross-Pitaevskii versus many-body dynamics; (x) Variance of an anisotropic BEC; and (xi) excitation spectrum of a weakly-interacting rotating BEC showing enhanced many-body effects. These are all basic and appealing, sometimes unexpected many-body results put forward with the generous allocation of computer time by the HLRS to the MCTDHB project. Finally, expected future developments and research tasks are prescribed, too.

O. E. Alon, V. S. Bagnato, R. Beinke, S. Basu, L. S. Cederbaum, B. Chakrabarti, B. Chatterjee, R. Chitra, F. S. Diorico, S. Dutta, L. Exl, A. Gammal, S. K. Haldar, S. Klaiman, C. Lévêque, R. Lin, N. J. Mauser, P. Molignini, L. Papariello, R. Roy, K. Sakmann, A. I. Streltsov, G. D. Telles, M. C. Tsatsos, R. Wu, A. U. J. Lode

Molecules, Interfaces, and Solids


Mesoscale Simulation of Dislocation Microstructures at Internal Interfaces

The need for predicting the behavior of crystalline materials on small-scales has led to the development of physically based descriptions of the motion of dislocations. Several dislocation-based continuum theories have been introduced, but only recently rigorous techniques have been developed for performing meaningful averages over systems of moving, curved dislocations, yielding evolution equations based on a dislocation density tensor. Those evolution equations provide a physically based framework for describing the motion of curved dislocations in three-dimensional systems. However, a meaningful description of internal interfaces and the complex mechanistic interaction of dislocations and interfaces in a dislocation based continuum model is still an open task. In this paper, we address the conflict between the need for a mechanistic modeling of the involved physical mechanisms and a reasonable reduction of model complexity and numerical effort. We apply a dislocation density based continuum formulation to systems which are strongly affected by internal interfaces, i.e. grain boundaries and interfaces in composite materials, and focus on the physical and numerical realization. Particularly, the interplay between physical and numerical accuracy is pointed out and discussed.

Katrin Schulz, Markus Sudmanns

Accurate and Efficient Spin-Spin Zero-Field Splitting Calculations for Extended Periodic Systems

Spin-spin zero-field splitting (ZFS) is a sensitive spectroscopic signature accessable by electron paramagnetic resonance. It is the key fingerprint used for the identification of high-spin defect centers in semiconducting host materials and for the characterization of their electronic structure. In recent years, much progress has been made in developing an efficient first-principles methodology for ZFS calculations that help in the interpretation of experimental data. Here we address the negatively charged nitrogen-vacancy center (NV $$^-$$ ) in diamond. It is used as a test system to explore the accuracy and efficiency of spin-spin zero-field splitting calculations on massively parallel computing systems.

T. Biktagirov, C. Braun, S. Neufeld, U. Gerstmann, W. G. Schmidt

Enhanced Acid Dissociation at the Solid/Liquid Interface

In this chapter we review some recent results from first principles molecular dynamics simulations which show how molecular properties, such as proton dissociation, can be influenced upon adsorption at a solid/liquid interface. In particular, we discuss in details the increased acidity of pyruvic acid at the quartz /water interface, which is of relevance for the chemistry of the atmosphere. Our simulations unveil the special role of the microsolvation at interface, as well as the role of the silanols in stabliziing the deprotonated form of the acid. The enhanced acidity at the hydrophilic quartz/water interface is at odd with what typically found at the water/air interface where acidity is normally reduced, and the associated form of the acidity stabilized.

Dominika Lesnicki, Marialore Sulpizi

Inorganic and Organic Functionalisation of Silicon Studied by Density Functional Theory

The present report deals with different approaches to functionalize the silicon surface studied by density functional theory. With the examples of the adsorption of tert-butylphosphine (TBP) on a silicon surface and the prediction of the band gap for the diluted nitride system Ga(N, As), elemental questions in the growth of inorganic semiconductors via the chemical vapour deposition (CVD) technique are addressed. The organic functionalisation is presented on the basis of the monofunctional cyclooctyne and its bifunctional derivate. Whereby, the adsorption dynamics and the lifetime of intermediate structures are studied with the aid of ab initio molecular dynamic simulations. The scaling of large simulations on the resources of the High-Performance Computing Center Stuttgart is addressed in a final section.

Fabian Pieck, Lisa Pecher, Jan-Niclas Luy, Ralf Tonner

Progress Report on: Sulfur in Ethylene Epoxidation on Silver (SEES2)

The primary goal of the “Sulfur in ethylene epoxidation on silver” (SEES2) project is to elucidate the mechanism(s) by which catalytic ethylene epoxidation occurs over silver surfaces. There is a particular focus on the role of sulfur. The program involves using density functional theory to predict stable surface phases under various conditions by way of ab initio atomistic thermodynamics. Once identified, the spectroscopic properties of candidate phases are computed to enable experimental verification. Minimum energy paths associated with the (re)formation and reaction of the identified phases are then computed to determine their possible roles in ethylene epoxidation. Through this approach we identified a novel $$\text {Ag(SO}_4$$ ) phase and showed it selectively transfers oxygen to ethylene to form the epoxide during temperature programed reaction. In the last year we have shifted the focus to the behavior of surface species under catalytic conditions, focusing on the 0 K minimum energy paths of the surface reaction network before moving on to finite temperature effects. In this effort we have identified additional phases and studied the competition between them. Of the studied species the novel $$\text {SO}_4$$ phase, where sulfur is present as S(V+), is the only silver one capable of selectively reacting with ethylene to form the epoxide. We further found the $$\text {SO}_4$$ species is rapidly regenerated through reaction with oxygen, which suppresses the coverage of an adsorbed $$\text {SO}_3$$ that appears to be selective in total oxidation. The presence of $$\text {SO}_x$$ species is also found to reduce the EO:AcH branching ratio associated with the reaction of ethylene with atomic oxygen on the unreconstructed Ag(111). Thus, it appears under conditions that are not artificially clean EO is produced in large part by oxygen transfer from the novel $$\text {SO}_4$$ phase. These new insights are only possible due to the use of various levels of parallelization to extend the scaling of our code on the Cray XC40 system Hazel Hen, which has allowed us to compute the minimum energy paths of a complex network of surface reactions.

Travis Jones

Reactive Flows


Towards Clean Propulsion with Synthetic Fuels: Computational Aspects and Analysis

In order to support sustainable powertrain concepts, synthetic fuels show significant potential to be a promising solution for future mobility. It was found that $$\mathrm {CO_2}$$ emissions during the combustion process of synthetic fuels can be reduced compared to conventional fuels and that sustainable fuel production pathways exists. Furthermore, it is possible to burn some synthetic fuels soot-free, which indirectly also eliminates the well-known soot- $$\mathrm {NO}_x$$ tradeoff. However, in order to use the full potential of the new fuels, optimization of currently used injection systems needs to be performed. This is still challenging since fundamental properties are not known and pollutant formation is a multi-physics, multi-scale process. Therefore, the high-fidelity simulation framework CIAO is improved and optimized for predictive simulations of multiphase, reactive injections in complex geometries. Due to the large separation of scales, these simulations are only possible with current supercomputers. This work discusses the computational performance of the high-fidelity simulations especially focusing on vectorization, scaling, and input/output (I/O) on Hazel Hen (Cray XC40) supercomputer at the High Performance Computing Center Stuttgart (HLRS). Moreover, the impact of different internal nozzle flow initial conditions is shown, the effect of different chemical mechanisms studied, and the predictability of soot emissions investigated. The Spray A case defined by the Engine Combustion Network (ECN) is used as the target case due to the availability of experimental data for this injector.

Mathis Bode, Marco Davidovic, Heinz Pitsch

Improved Vectorization for Efficient Chemistry Computations in OpenFOAM for Large Scale Combustion Simulations

The computation of chemical reaction rates is commonly the performance bottleneck in CFD simulations of turbulent combustion with detailed chemistry. Therefore, an optimization method is used where C++ source code is automatically generated for arbitrary reaction mechanisms. The generated code is highly optimized for the chosen mechanism and contains all routines for computing chemical reaction rates. In this work, the serial performance of the automatically generated source code, which in an earlier work only used ISO C++, is further improved by utilizing two compiler extensions: restrict and __builtin_assume_aligned. Introducing these two extensions to the generated code reduces the time for computing chemical reaction rates by up to 50% for the investigated reaction mechanism and total simulation time by up to 25%. Compared to OpenFOAM’s standard chemistry implementation, the new code is faster by a factor of 10. This work discusses the effect of the two compiler extensions on performance by looking at two specific kernel functions from the automatically generated code and the effect on the assembly generated by the gcc and Intel compilers. The newly optimized code is used to evaluate the performance gain in a large scale parallel case, which simulates an experimentally investigated turbulent flame of laboratory scale on 14,400 CPU cores on the Hazel Hen cluster at HLRS. In the simulation, no combustion models are used and the flame is resolved down to the smallest length scales. With this approach, comparison of measured data with the simulation shows very good agreement. Using the optimized code including compiler extensions, total simulation time decreases by 20% compared to the same code without compiler extensions. A comprehensive database from the simulation results has been assembled and will consist of 10 TB of 3D and 2D transient field variables.

Thorsten Zirwes, Feichi Zhang, Jordan A. Denev, Peter Habisreuther, Henning Bockhorn, Dimosthenis Trimis

Computational Fluid Dynamics


DNS of Compressible Turbulent Boundary Layers with Adverse Pressure Gradients

First direct-numerical-simulation results of compressible subsonic adverse-pressure-gradient turbulent boundary-layers are presented with self-similar boundary-layer profiles in their streamwise evolution, which represents the most general canonical form of the adverse pressure gradient case. Only few results are available in literature for this problem even in the incompressible regime, since the achievement of such canonical flows requires very costly iterative procedures in order to find the correct pressure distribution which has to be prescribed at the top of the simulation domain. Additionally, preliminary result are presented for the detection of turbulent superstructures in the unsteady flow field, which can be denoted to be the most dominant global events in wall-bounded turbulent flows. All results have been calculated with a new version of our numerical in-house code NS3D, which now also allows MPI decomposition in the spanwise direction of the simulation domain. A scaling study shows a maximum increase in the efficiency of up to 300% compared to the previous code version where only OpenMP decomposition has been available for the spanwise decomposition.

Christoph Wenzel, Johannes M. F. Peter, Björn Selent, Matthias B. Weinschenk, Ulrich Rist, Markus J. Kloker

Towards a Direct Numerical Simulation of Primary Jet Breakup with Evaporation

The primary breakup of liquid jets plays an important role in fuel injection for combustion engines and gas turbines. Due to the ambient conditions the liquid also evaporates during breakup. Direct Numerical Simulations (DNS) are well suited for the analysis of this phenomena. As a first step towards an understanding of this problem, a DNS of an evaporating jet is carried out and a comparison with a non evaporating jet is presented. We use the in-house 3D Computational Fluid Dynamics (CFD) code Free Surface 3D (FS3D) to solve the incompressible Navier-Stokes equations. The Volume of Fluid (VOF) method is used in combinations with Piecewise Linear Interface Calculation (PLIC) to reconstruct a sharp interface. The energy equation is solved to obtain the phase change at the interface. We were able to conduct a purely Eulerian simulation of an evaporating jet during atomization. The morphology of the liquid jets and the vapour concentration are shown and analysed. The droplet size distribution shows the influence of evaporation, leading to smaller droplets. The presented results are in good accordance with our expectation as well as with other investigations in the literature. In addition, we present the results of an investigation into replacing pointers to fields in our code with arrays to improve computational performance. With this change we are able to increase the efficiency of the code and obtain a speed up of more than 40%.

Jonathan Reutzsch, Moritz Ertl, Martina Baggio, Adrian Seck, Bernhard Weigand

Direct Numerical Simulation of Turbulent Flow Past an Acoustic Cavity Resonator

The present project studies the sound pressure inside an acoustic resonant cavity driven by turbulent flow. With this numerical study as an interim result, the ultimate goal is to improve the industrial layout of resonant cavities, in general: For example to circumvent resonance conditions. The current rise of high-performance computing allows us to simulate the dynamics of the non-linear turbulence-acoustic interaction by the high-quality method of a Direct Numerical Simulation (DNS). For the first time, the three dimensional geometry investigated here can be studied numerically in full detail without simplification. So far numerical studies of Helmholtz resonators with resolved neck shape do not consider an inflowing turbulent boundary layer or do not resolve all system scales, but assume some form of turbulence model. To effectively run the DNS on a supercomputer, a multi-block parallelization method is newly implemented for complex geometries, which consist of multiple, different sized blocks. Both in strong and weak scaling test a previous single-block parallelization is outperformed. The optimal load of gridpoints per core is identified and a distinction between misleading and meaningful weak scaling tests is made.

Lewin Stein, Jorn Sesterhenn

High-Fidelity Direct Numerical Simulation of Supercritical Channel Flow Using Discontinuous Galerkin Spectral Element Method

Supercritical fluids are suggested as one of the potential candidates for the next generation nuclear reactor by Generation IV nuclear forum to improve the thermal efficiency. But, supercritical fluids suffer from the deteriorated heat transfer under certain conditions. This deteriorated heat transfer phenomenon is a result of a peculiar attenuation of turbulence within the flow. This peculiarity is difficult to predict by conventional turbulence modeling. Therefore, direct numerical simulations were used in the past employing the low-Mach assumption with a finite volume code. As a next step, we extend the discontinuous Galerkin spectral element method for direct numerical simulation of supercritical carbon dioxide. The higher-order of accuracy and fully compressible code improve the fidelity of the simulations. A computationally robust and efficient implementation of the equation of state was used which is based on adaptive mesh refinement. The objective of this report is to demonstrate the usage of the code in complex flow. Therefore, channel geometry is adopted and simulations were conducted at different Mach numbers to observe the effects of compressibility in the supercritical fluid regime. The isothermal boundary conditions were used at the walls of the channel. The mean profile of pressure, density and temperature are drastically affected by the Mach number variation. In the end, scalability tests were conducted and code shows a very good parallel scalability up to 12,000 cores.

Fabian Föll, Sandeep Pandey, Xu Chu, Claus-Dieter Munz, Eckart Laurien, Bernhard Weigand

Application and Development of the High Order Discontinuous Galerkin Spectral Element Method for Compressible Multiscale Flows

This paper summarizes our progress in the application and feature development of a high-order discontinuous Galerkin (DG) method for scale resolving fluid dynamics simulations on the Cray XC40 Hazel Hen cluster at HLRS. We present the extension to Chimera grid techniques which allow efficient computations on flexible meshes, and discuss data-based development of subgrid closure terms through machine learning algorithms. We also show the first results of the simulation of a challenging, high Reynolds number supersonic dual nozzle flow.

Andrea Beck, Thomas Bolemann, David Flad, Nico Krais, Jonas Zeifang, Claus-Dieter Munz

Application of the SPH Method to Predict Primary Breakup in Complex Geometries

Understanding the process of primary breakup of liquids of air-assisted atomization systems is of major importance for the optimization of spray nozzles. Experimental investigations face various limitations. Conventional numerical methods for the simulation of the process are subject of various shortcomings. In the present paper the “Smoothed Particle Hydrodynamics” (SPH)-method and its advantages over conventional methods are presented. The suitability of the method is demonstrated by analyzing two different air-assisted atomization systems. First, a methodology for the investigation of the two-phase flow in fuel spray nozzles for aero-engine combustors is introduced and predictions of the flow in a typical nozzle geometry are presented. Second, numerical simulations of the atomization of a viscous slurry, which is used in gasifiers for biofuel production, are presented and compared to experimental results. Highly valuable results are retrieved, which will improve the fundamental understanding of primary breakup and will enable to optimize nozzle geometries.

G. Chaussonnet, T. Dauch, S. Braun, M. Keller, J. Kaden, C. Schwitzke, T. Jakobs, R. Koch, H. -J. Bauer

Enhancement and Application of the Flow Solver FLOWer

Recent enhancements and applications of the flow solver FLOWer are presented in this paper. A locally formulated laminar-turbulent transition model is implemented and used for simulations of a steady and pitching finite wing. Corresponding experimental results show good agreement on transition points. In CFD, a separation bubble is observed that triggers laminar-turbulent transition. Furthermore, the paper focuses on CFD simulations conducted on Airbus Helicopters’ compound helicopter RACER and the underlying setup, including flexible main rotor blades, engine boundary conditions and deformable flaps. Finally, interaction phenomena and interesting flow characteristics are described for hovering and cruise conditions.

Felix Frey, Johannes Herb, Johannes Letzgus, Pascal Weihing, Manuel Keßler, Ewald Krämer

CFD Simulations in Complex Nuclear Containment Configurations

Two-phase flows with water droplets have a significant influence on the thermal-hydraulic behaviour within Pressurized Water Reactors PWR. Such flows take place in the form of spray cooling, inter alia, in French nuclear reactors. In the case of a leak in the primary circuit of a PWR, hot steam will be released in the containment, which results in a pressure and temperature increase. The spray system ensures the reduction of the containment pressure and may, thus, help to avoid nuclear incidents. This work presents firstly a CFD simulation of spray cooling as well as aerosol particle washout by means of a spray system in a real size nuclear containment. The parallel performance of the simulations within a two-room model containment called $$THAI^+$$ is also investigated. Due to the large size and geometric complexity of this configuration, numerical grids with high refinement levels have to be generated to get accurate simulation results. For this reason, a good scalable CFD code is indispensable in order to achieve accurate simulations in realistic computional times. Since another aim of this work is to define guidelines for the optimum use of computational resources, the scalability tests will be performed on four different grid styles with six mesh resolutions up to $$39{-}40\times 10^6$$ elements.

A. Mansour, C. Kaltenbach, E. Laurien

Cavitation Simulations of a Tip Leakage Vortex for a NACA0009 Hydrofoil and a Francis Turbine at Stable Full Load Operating Point

In the first part of the study, the ability of homogeneous and inhomogeneous two-phase modeling approach is investigated using a NACA0009 hydrofoil with a cavitating tip leakage vortex. The results indicate that the inhomogeneous model does not increase accuracy of the simulation results. This can be explained by the small diameter of the cavitation bubbles, which results in a strong coupling between liquid and vapor phase. Based on the results for the test case, homogeneous model is applied for two-phase simulations of a Francis turbine at a stable full load operating point. The mesh study demonstrates the need for fine meshes with a size of approximately 50 million elements. While the effect of the cavitation constants can be neglected, the geometry of the runner nut and enabling curvature correction in the SST turbulence model has a relevant impact on simulation accuracy. The use of a transient rotor stator interface is challenging for computational performance. Applying multipass partitioning method increases the speedup by around 10%. Further performance improvement can be achieved with the expert parameter $$parallel \; optimization \; level$$ . All in all, a parallelization on 1200 cores is reasonable.

Jonas Wack, Stefan Riedelbauch

Transport and Climate


Climate Change Studies for Germany and Europe Using High Resolution WRF Simulations

During the 21 $$\mathrm{{st}}$$ century, not only the global mean surface temperature but also the amount and strength of weather extremes will increase. This is mainly due to the continuous emission of greenhouse gases by human activities. Since warming over many land areas is larger than over the oceans, climate conditions will be regionally highly variable. In order to make regional projections of the climate in the future and to evaluate the ability of models to represent such phenomena, multi-model ensembles of regional climate simulations are required. This is e.g. realized in the World Climate Research Program initiated COordinated Regional climate Downscaling EXperiment CORDEX. The objective of WRFCLIM at HLRS is to contribute to these multi-model ensembles with simulations based on the Weather and Research Forecasting (WRF) model. Five simulations were carried out within the framework of the BMBF funded Project ReKliEs-De (Regionale Klimasimulationen Ensemble für Deutschland) for the simulation time period 1958–2100 at 12 km resolution. These simulations represent the first WRF climate projections which have been realized on the HLRS supercomputer yet. The results demonstrate that the ensemble members provided by the WRF provide an excellent contribution to the spread of the ReKliEs-De ensemble results. This gives a much better confidence not only in the estimation of the evolution of medians of atmospheric variables due to climate change but also in the estimation of extreme values such as hot and ice days. The corresponding climate indices provided by ReKliEs-De underline the importance for society and economy to mitigate climate change. Furthermore, within the CORDEX Flagship Pilot Study framework simulations are carried out between 1999 and 2014 at 15 km resolution further downscaling to the convection permitting (CP) grid on 3 km in central Europe to assess mainly diurnal cycles and high impact weather. The results and the performance analyses on the CP scale will be fundamental for to the set up and improvement of the next generation climate simulations. Therefore, HLRS and UHOH are prepared to contribute to a series of national and international projects concerning seasonal and climate simulations. The results highlight the necessity of CP simulations for next generation earth system models, and multi-model ensembles to assess climate change, as the latter provide indispensable uncertainty measures for climate change signals and extremes for Germany.

Kirsten Warrach-Sagi, Viktoria Mohr, Josipa Milovac, Thomas Schwitalla, Volker Wulfmeyer

Seasonal Simulation of Weather Extremes (WRFXXXL)

To date, seasonal forecasts are often performed by applying horizontal resolutions of 75–150 km due to lack of computational resources and associated operational constraints. As this resolution is too coarse to represent fine scale structures impacting the large scale circulation, a convection permitting (CP) resolution of less than 4 km horizontal resolution is required. Most of the simulations are carried out as limited area model (LAMs) and thus require boundary conditions at all four domain boundaries. In this study, the Weather Research and Forecasting (WRF) model is applied in a latitude belt set-up in order to avoid the application of zonal boundaries. The horizontal resolution is 0.03 $$^{\circ }$$ and 0.45 $$^{\circ }$$ spanning a belt between 65 $$^{\circ }$$ N and 57 $$^{\circ }$$ S encompassing 12000 * 4060 * 57 grid boxes for the high-resolution domain. The simulations are driven by ECMWF analysis and high resolution SST data from multiple sources. The simulation period was February 1st–July 1st, 2015 which was a strong El Niño period.

Thomas Schwitalla, Volker Wulfmeyer, Kirsten Warrach-Sagi

Computer Science


PetaFLOP Molecular Dynamics for Engineering Applications

Molecular dynamics (MD) simulations enable the investigation of multicomponent and multiphase processes relevant to engineering applications, such as droplet coalescence or bubble formation. These scenarios require the simulation of ensembles containing a large number of molecules. We present recent advances within the MD framework ls1 mardyn which is being developed with particular regard to this class of problems. We discuss several OpenMP schemes that deliver optimal performance at node-level. We have further introduced nonblocking communication and communication hiding for global collective operations. Together with revised data structures and vectorization, these improvements unleash PetaFLOP performance and enable multi-trillion atom simulations on the HLRS supercomputer Hazel Hen. We further present preliminary results achieved for droplet coalescence scenarios at a smaller scale.

Philipp Neumann, Nikola Tchipev, Steffen Seckler, Matthias Heinen, Jadran Vrabec, Hans-Joachim Bungartz

Load-Balancing and Spatial Adaptivity for Coarse-Grained Molecular Dynamics Applications

We present our approach for a scalable implementation of coupled soft matter simulations for inhomogeneous applications based on the simulation package ESPResSo and an extended version of the adaptive grid framework p4est. Our main contribution in this paper is the development and implementation of a joint partitioning of two or more distinct octree-based grids based on the concept of a finest common tree. This concept guarantees that, on all grids, the same process is responsible for each point in space and, thus, avoids communication of data in overlapping volumes handled in different partitions. We achieve up to 85% parallel efficiency in a weak scaling setting. Our proposed algorithms take only a small fraction of the overall runtime of grid adaption.

Steffen Hirschmann, Michael Lahnert, Carolin Schober, Malte Brunn, Miriam Mehl, Dirk Pflüger

Miscellaneous Topics


Scalability of a Parallel Monolithic Multilevel Solver for Poroelasticity

This study investigates a solver for the quasi-static Biot model for soil consolidation. The scheme consists of an extrapolation scheme in time, complemented by a scalable monolithic multigrid method for solving the linear systems resulting after spatial discretisation. The key ingredient is a fixed-stress inexact Uzawa smoother that has been suggested and analysed using local Fourier analysis before (Gaspar and Rodrigo, Comput Methods Appl Mech Eng 326:526–540, 2017, [8]). The work at hand investigates the parallel properties of the resulting multigrid solver. For a 3D benchmark problem with roughly 400 million degrees of freedom, scalability is demonstrated in a preliminary study on Hazel Hen. The presented solver framework should be seen as a prototype, and can be extended and generalized, e.g., to non-linear problems easily.

Arne Nägel, Gabriel Wittum

High Fidelity Monte Carlo for Fusion Neutronics

The High Fidelity Monte Carlo for fusion neutronics project HIFIMC aims at providing the computational resources for the development, testing and application of advanced modeling and simulation techniques as required for the design and optimization of upcoming fusion reactors like ITER, DEMO, HELIAS and related research facilities like IFMIF or DONES. Large-scale simulations are required to provide high fidelity results to assess the performance of such facilities. These include numerous time consuming Monte Carlo simulations for describing the transport of particles (neutrons and gammas) through the complex and heterogeneous geometry. The main computational tool for such application is the MCNP Monte Carlo code, developed at LANL, US, and the coupled transport-activation tool R2Smesh, developed at KIT. Several concurrent research topics have been conducted and are reported here. Further development of R2Smesh on enhancing performance and studies on convergence issues in meshing of the activation responses are required to qualify the tools for many large nuclear performance and radiation shielding applications in fusion devices. Verification and validation of alternative codes, like GEANT 4, developed at CERN, CH, are supporting these efforts. Applications to port systems in ITER (next step fusion reactor under construction in France), breeding blankets in DEMO (Demonstration fusion reactor), full reactor of HELIAS (Helical-Axis Advanced Stellarator) and irradiation test systems of IFMIF/DONES (International Fusion Material Irradiation Facility/DEMO-oriented neutron source) are presented. It is shown that the tools on high-performance computing platforms are capable to tackle the challenging problems of radiation shielding and activation in complex geometries involving both deep penetration and radiation streaming.

B. Weinhorst, D. Leichtle, A. Häußler, E. Nunnenmann, P. Pereslavtsev, Y. Qiu, P. Raj, A. Travleev, U. Fischer

Molecular Modeling and Simulation: Force Field Development, Evaporation Processes and Thermophysical Properties of Mixtures

To gain physical insight into the behavior of fluids on a microscopic level as well as to broaden the data base for thermophysical properties especially for mixtures, molecular modeling and simulation is utilized in this work. Various methods and applications are discussed, including a procedure for the development of new force field models. The evaporation of liquid nitrogen into a supercritical hydrogen atmosphere is presented as an example for large scale molecular dynamics simulation. System-size dependence and scaling behavior are discussed in the context of Kirkwood-Buff integration. Further, results for thermophysical mixture properties are presented, i.e. the Henry’s law constant of aqueous systems and diffusion coefficients of a ternary mixture.

Tatjana Janzen, Robin Fingerhut, Matthias Heinen, Andreas Köster, Y. Mauricio Muñoz-Muñoz, Jadran Vrabec

Multiphase-Field Modeling and Simulation of Martensitic Phase Transformation in Heterogeneous Materials

The martensitic phase transformation is an important mechanism which strongly changes the properties of the material. On the one hand, martensite is a widely used and purposefully produced constituent of many high-strength steels. On the other hand, martensite can also lead in some cases to an unwanted embrittlement of the material, so that this transformation has to be prevented.

E. Schoof, C. Herrmann, D. Schneider, J. Hötzer, B. Nestler

KKRnano: Quantum Description of Skyrmions in Chiral B20 Magnets

We present the latest version of the linear-scaling electronic structure code KKRnano, in which an enhanced Korringa-Kohn-Rostoker (KKR) scheme is utilized to perform Density Functional Theory (DFT) calculations. The code allows us to treat system sizes of up to several thousands of atoms per unit cell and to simulate a non-collinear alignment of atomic spins. This capability is used to investigate nanometer-sized magnetic textures in the germanide B20-MnGe, a material that is potentially going to play an important role in future spintronic devices. A performance analysis of KKRnano on Hazel Hen emphasizes the good scaling behaviour with increasing system size and demonstrates the extensive integration of highly optimized libraries.

Marcel Bornemann, Paul F. Baumeister, Rudolf Zeller, Stefan Blügel

A Multi-resolution Approach to the Simulation of Protein Complexes in a Membrane Bilayer

We describe a transferable multiresolution computational approach to build and simulate complexes of two proteins—cytochrome P450 (CYP) and CYP reductase (CPR)—in a membrane bilayer using Brownian dynamics (BD) and all-atom molecular dynamics (MD) simulations. Our benchmarks showed that MD simulations of these systems could be carried out efficiently with up to 180 nodes (4320 cores) using NAMD version 2.12. Our results provide a basis for defining the ensemble of electron transfer-competent arrangements of CYP-CPR-membrane complexes and for understanding differences in the interactions with CPR of different CYPs, which have implications for CYP-mediated drug metabolism and the exploitation of CYPs as drug targets. This work was carried out in the DYNATHOR (DYNAmics of THe complex of cytOchrome P450 and cytochrome P450 Reductase in a phospholipid bilayer) project at HLRS.

Goutam Mukherjee, Prajwal Nandekar, Ghulam Mustafa, Stefan Richter, Rebecca C. Wade
Additional information

Premium Partner

    Image Credits