Skip to main content

Advertisement

SpringerLink
Log in

A distributed parallel multiple-relaxation-time lattice Boltzmann method on general-purpose graphics processing units for the rapid and scalable computation of absolute permeability from high-resolution 3D micro-CT images

  • Original paper
  • Published:
Computational Geosciences Aims and scope Submit manuscript

Abstract

Digital rock physics (DRP) is a rapidly evolving technology targeting fast turnaround times for repeatable core analysis and multi-physics simulation of rock properties. We develop and validate a rapid and scalable distributed-parallel single-phase pore-scale flow simulator for permeability estimation on real 3D pore-scale micro-CT images using a novel variant of the lattice Boltzmann method (LBM). The LBM code implementation is designed to take maximum advantage of distributed computing on multiple general-purpose graphics processing units (GPGPUs). We describe and extensively test the distributed parallel implementation of an innovative LBM algorithm for simulating flow in pore-scale media based on the multiple-relaxation-time (MRT) model that utilizes a precise treatment of body force. While the individual components of the resulting simulator can be separately found in various references, our novel contributions are (1) the integration of all of the mathematical and high-performance computing components together with a highly optimized code implementation and (2) the delivery of quantitative results with the simulator in terms of robustness, accuracy, and computational efficiency for a variety of flow geometries including various types of real rock images. We report on extensive validations of the simulator in terms of accuracy and provide near-ideal distributed parallel scalability results on large pore-scale image volumes that were largely computationally inaccessible prior to our implementation. We validate the accuracy of the MRT-LBM simulator on model geometries with analytical solutions. Permeability estimation results are then provided on large 3D binary microstructures including a sphere pack and rocks from various sandstone and carbonate formations. We quantify the scalability behavior of the distributed parallel implementation of MRT-LBM as a function of model type/size and the number of utilized GPGPUs for a panoply of permeability estimation problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adler, P.M., Jacquin, C.G., Quiblier, J.A.: Flow in simulated porous media. Intern. J. Multiphase Flow 16, 691–712 (1990)

    Article  Google Scholar 

  2. Andrä, H., Combaret, N., Dvorkin, J., Glatt, E., Han, J., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E.H., Sain, R., Saxena, N., Ricker, S., Wiegmann, A., Zhan, X.: Digital rock physics benchmarks—part I: imaging and segmentation. Comput. Geosci. 50, 25–32 (2013)

    Article  Google Scholar 

  3. Andrä, H., Combaret, N., Dvorkin, J., Glatt, E., Han, J., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E.H., Sain, R., Saxena, N., Ricker, S., Wiegmann, A., Zhan, X.: Digital rock physics benchmarks—part II: computing effective properties. Comput. Geosci. 50, 33–43 (2013)

    Article  Google Scholar 

  4. ANSYS Inc: ANSYS FLUENT Theory Guide (2010)

  5. Arns, C.H., Bauget, F., Limaye, A., Sakellariou, A., Senden, T., Sheppard, A., Sok, R.M., Pinczewski, V., Bakke, S., Berge, L.I., Oren, P., Knackstedt, M.: Pore-scale characterization of carbonates using x-ray microtomography. SPE J. 10, 26–29 (2005)

    Article  Google Scholar 

  6. Arns, C.H., Knackstedt, M.A., Pinczewski, W.V., Garboczi, E.J.: Computation of linear elastic properties from microtomographic images: methodology and agreement between theory and experiment. Geophysics 67, 1396–1405 (2002)

    Article  Google Scholar 

  7. Bear, J.: Dynamics of fluids in porous media. American Elsevier Publishing Company Inc., New York (1972)

    Google Scholar 

  8. Benzi, R., Succi, S., Vergassola, M.: The lattice Boltzmann-equation—theory and applications. Phys. Rep. 222, 145–197 (1992)

    Article  Google Scholar 

  9. Berg, S., Ott, H., Klapp, S.A., Schwing, A., Neiteler, R., Brussee, N., Makurat, A., Leu, L., Enzmann, F., Schwarz, J. -O., Kersten, M., Irvine, S., Stampanoni, M.: Real-time 3D imaging of Haines jumps in porous media flow. Proc. Nat. Acad. Sci. (PNAS) 110, 3755–3759 (2013)

    Article  Google Scholar 

  10. Berg, S., Rücker, M., Ott, H., Georgiadis, A., van der Linde, H., Enzmann, F., Kersten, M., Armstrong, R.T., de With, S., Becker, J., Wiegmann, A.: Connected pathway relative permeability from pore-scale imaging of imbibition. Adv. Water Res. 90, 24–35 (2016)

    Article  Google Scholar 

  11. Bernaschi, M., Fatica, M., Melchionna, S., Succi, S., Kaxiras, E.: A flexible high-performance lattice Boltzmann GPU code for the simulations of fluid flows in complex geometries. Concurr. Comput.: Prac. Exp. 22, 1–14 (2010)

    Article  Google Scholar 

  12. Berryman, J.G., Blair, S.C.: Use of digital image analysis to estimate fluid permeability of porous materials: application of two-point correlation functions. J. Appl. Phys. 60, 1930–1938 (1986)

    Article  Google Scholar 

  13. Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component systems. Phys. Rev. 94, 511–525 (1954)

    Article  Google Scholar 

  14. Bisson, M., Bernaschi, M., Melchionna, S., Succi, S., Kaxiras, E.: Multiscale hemodynamics using GPU clusters. Commun. Comput. Phys. 11, 48–64 (2012)

    Article  Google Scholar 

  15. Blunt, M.J., Bijeljic, B., Dong, H., Gharbi, O., Iglauer, S., Mostaghimi, P., Paluszny, A., Pentland, C.: Pore-scale imaging and modelling. Adv. Water Res. 51, 197–216 (2013)

    Article  Google Scholar 

  16. Bouzidi, M.H., Firdaouss, M., Lallemand, P.: Momentum transfer of a Boltzmann-lattice fluid with boundaries. Phys. Fluids 13, 3452–3459 (2001)

    Article  Google Scholar 

  17. Cancelliere, A., Chang, C., Foti, E., Rothman, D.H., Succi, S.: The permeability of a random medium: comparison of simulation with theory. Phys. Fluids A 2, 2085–2088 (1990)

    Article  Google Scholar 

  18. Chen, H., Teixeira, C., Molvig, K.: Digital physics approach to computational fluid dynamics: some basic theoretical features. Intern. J. Modern Phys. C 8, 675–684 (1997)

    Article  Google Scholar 

  19. Chen, C., Wang, Z., Majeti, D., Vrvilo, N., Warburton, T., Sarkar, V., Li, G.: Optimization of lattice Boltzmann simulation with graphics-processing-unit parallel computing and the application in reservoir characterization. SPE J. 21(3), 1425–1435 (2016)

    Article  Google Scholar 

  20. COMSOL Multiphysics Documentation: COMSOL Multiphysics, version 5.3, Pore-scale flow. https://www.comsol.com/model/download/407771/models.ssf.porescaleflow.pdf. Accessed: 4 June 2017 (2017)

  21. d’Humières, D., Ginzburg, I., Krafczyk, M., Lallemand, P., Luo, L.: Multiple-relaxation-time lattice Boltzmann models in three dimensions. Philos. Trans. Royal Soc. A 360, 437–451 (2002)

    Article  Google Scholar 

  22. Dvorkin, J., Armbruster, M., Baldwin, C., Fang, Q., Derzhi, N., Gomez, C., Nur, B., Nur, A., Mu, Y.: The future of rock physics: computational methods vs. lab testing. First Break 26, 63–68 (2008)

    Google Scholar 

  23. Dvorkin, J., Derzhi, N., Diaz, E., Fang, Q.: Relevance of computational rock physics. Geophysics 76, E141–E153 (2011)

    Article  Google Scholar 

  24. Ferrèol, B., Rothman, D.H.: Lattice-Boltzmann simulations of flow through Fontainebleau sandstone. Transp. Porous Media 20, 3–20 (1995)

    Article  Google Scholar 

  25. Ginzburg, I., d’Humières, D.: Multireflection boundary conditions for lattice Boltzmann models. Phys. Rev. E 68, 066614 (2003)

    Article  Google Scholar 

  26. Gray, F., Boek, E.: Enhancing computational precision for lattice Boltzmann schemes in porous media flows. Computation 4, 11 (2016)

    Article  Google Scholar 

  27. Guo, Z., Zheng, C.: Analysis of lattice Boltzmann equation for microscale gas flows: relaxation times, boundary conditions and the Knudsen layer. Intern. J. Comput. Fluid Dyn. 22, 465–473 (2008)

    Article  Google Scholar 

  28. Guo, Z., Zheng, C., Shi, B.: Discrete lattice effects on the forcing term in the lattice Boltzmann method. Phys. Rev. E 65, 046308 (2002)

    Article  Google Scholar 

  29. Hassanizadeh, S.M., Gray, W.: High velocity flow in porous media. Transp. Porous Media 2, 521–531 (1987)

    Article  Google Scholar 

  30. He, X., Luo, L.S.: Theory of the lattice Boltzmann method: from the Boltzmann equation to the lattice Boltzmann equation. Phys. Rev. E 56, 6811–6817 (1997)

    Article  Google Scholar 

  31. He, X., Zou, Q., Luo, L., Dembo, M.: Analytic solutions of simple flows and analysis of nonslip boundary conditions for the lattice Boltzmann BGK model. J. Stat. Phys. 87, 115–136 (1997)

    Article  Google Scholar 

  32. Hinch, E.J.: An averaged-equation approach to particle interactions in a fluid suspension. J. Fluid Mech. 83, 695–720 (1977)

    Article  Google Scholar 

  33. Howells, I.D.: Drag due to the motion of a Newtonian fluid through a sparse random array of small fixed rigid objects. J. Fluid Mech. 64, 449–476 (1974)

    Article  Google Scholar 

  34. Keehm, Y.: Computational rock physics: Transport properties in porous media and applications. Ph.D. Dissertation, Stanford University, Stanford (2003)

    Google Scholar 

  35. Keehm, Y., Mukerji, T., Nur, A.: Permeability prediction from thin sections: 3D reconstruction and Lattice-Boltzmann flow simulation. Geophys. Res. Lett. 31, L04606 (2004)

    Article  Google Scholar 

  36. Kendon, V.M., Cates, M.E., Desplat, J. -C., Pagonabarraga, I., Bladon, P.: Inertial effects in three-dimensional spinodal decomposition of a symmetric binary fluid mixture: a lattice Boltzmann study. J. Fluid Mech. 440, 147–203 (2000)

    Google Scholar 

  37. Knackstedt, M.A., Sheppard, A.P., Sahimi, M.: Pore network modelling of two-phase flow in porous rock: the effect of correlated heterogeneity. Adv. Water Res. 24, 257–277 (2001)

    Article  Google Scholar 

  38. Knackstedt, M.A., Arns, C., Madadi, M., Sheppard, A.P., Latham, S., Sok, R., Bächle, G., Eberli, G.: Elastic and flow properties of carbonate core derived from 3D X ray-CT images. SEG Tech. Prog. Expanded Abstr. 27, 1804–1809 (2008)

    Google Scholar 

  39. Knackstedt, M.A., Latham, S., Madadi, M., Sheppard, A., Varslot, T., Arns, C.: Digital rock physics: 3D imaging of core material and correlations to acoustic and flow properties. Lead. Edge 28, 28–33 (2009)

    Article  Google Scholar 

  40. Kuzmin, A., Guo, Z., Mohamad, A.: Simultaneous incorporation of mass and force terms in the multi-relaxation-time framework for lattice Boltzmann schemes. Philos. Transac. Royal Soc. A Math. Phys. Eng. Sci. 369, 2219–2227 (2011)

    Article  Google Scholar 

  41. Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation: part 1: theoretical foundation. J. Fluid Mech. 271, 285–309 (1994)

    Article  Google Scholar 

  42. Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation: part 2. Numerical results. J. Fluid Mech. 271, 311–339 (1994)

    Article  Google Scholar 

  43. Lallemand, P., Luo, L.S.: Theory of the lattice Boltzmann method: dispersion, dissipation, isotropy, Galilean invariance, and stability. Phys. Rev. E 61, 6546–6562 (2000)

    Article  Google Scholar 

  44. Leu, L., Berg, S., Enzmann, F., Armstrong, R.T., Kersten, M.: Fast x-ray micro-tomography of multiphase flow in Berea Sandstone: a sensitivity study on image processing. Transp. Porous Media 105, 451–469 (2014)

    Article  Google Scholar 

  45. Llwellin, E.W.: LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part I: theory and implementation. Comput. Geosci. 36, 115–122 (2010)

    Article  Google Scholar 

  46. Llwellin, E.W.: LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part II: usage and validation. Comput. Geosci. 36, 123–132 (2010)

    Article  Google Scholar 

  47. Martys, N.S., Chen, H.: Simulation of multicomponent fluids in complex three dimensional geometries by the lattice Boltzmann method. Phys. Rev. E 53, 743–750 (1996)

    Article  Google Scholar 

  48. Mavko, G., Mukerji, T., Dvorkin, J.: The rock physics handbook: Tools for seismic analysis of porous media, 2nd edn. Cambridge University Press, Cambridge (2009)

    Book  Google Scholar 

  49. Meakin, P., Tartakovsky, A.M.: Modeling and simulation of pore-scale multiphase fluid flow and reactive transport in fractured and porous media. Rev. Geophys. RG3002, 1–47 (2009)

    Google Scholar 

  50. Otomo, H., Fan, H., Li, Y., Dressler, M., Staroselsky, I., Zhang, R., Chen, H.: Studies of accurate multi-component lattice Boltzmann models on benchmark cases required for engineering applications. J. Comput. Sci. 17, 334–339 (2016)

    Article  Google Scholar 

  51. Øren, P., Bakke, S., Rueslåtten, H.: Digital core laboratory: rock and flow properties derived from computer generated rocks. In: International Symposium of the Society of Core Analysts held in Trondheim, pp. 1–12, Norway (2006)

  52. Palabos Documentation. http://www.palabos.org/documentation/userguide/. Accessed: 4 June 2017 (2017)

  53. Pan, C., Luo, L.-S., Miller, C.T.: An evaluation of lattice Boltzmann schemes for porous medium flow simulation. Comput. Fluids 35, 898–909 (2006)

    Article  Google Scholar 

  54. Piller, M., Casagrande, D., Schena, G., Santini, M.: Pore-scale simulation of laminar flow through porous media. J. Phys.: Conf. Ser. 501, 012010 (2014)

    Google Scholar 

  55. Premnath, K.N., Abraham, J.: Three-dimensional multi-relaxation-time (MRT) lattice-Boltzmann models for multiphase flow. J. Comput. Phys. 224, 539–559 (2007)

    Article  Google Scholar 

  56. Premnath, K.N., Pattison, M.J., Banerjee, S.: An investigation of the Lattice Boltzmann method for large eddy simulation of complex turbulent separated flow. J. Fluids Eng. 135, 051401–1-12 (2013)

    Article  Google Scholar 

  57. Rumpf, H.C.H., Gupte, A.R.: Einflüsse der Porosität und Korngröarticleßenverteilung im Widerstandsgesetz der Porenströmung. Chem. Ing. Tech. 43, 367–375 (1971)

    Article  Google Scholar 

  58. Saenger, E.H., Enzmann, F., Keehm, Y., Steeb, H.: Digital rock physics: effect of fluid viscosity on effective elastic properties. J. Appl. Geophys. 74, 236–241 (2011)

    Article  Google Scholar 

  59. Sain, R., Mukerji, T., Mavko, G.: How computational rock-physics tools can be used to simulate geologic processes, understand pore-scale heterogeneity, and refine theoretical models. Lead. Edge 33, 324–334 (2014)

    Article  Google Scholar 

  60. Saxena, N., Mavko, G.: Estimating elastic moduli of rocks from thin sections: digital rock study of 3D properties from 2D images. Comput. Geosci. 88, 9–21 (2016)

    Article  Google Scholar 

  61. Saxena, N., Hofmann, R., Alpak, F.O., Berg, S., Dietderich, J., Agarwal, U., Tandon, K., Hunter, S., Freeman, J., Wilson, O.B.: References and benchmarks for pore-scale flow simulated using micro-CT images of porous media and digital rocks. Adv. Water Res. 109, 211–235 (2017)

    Article  Google Scholar 

  62. Saxena, N., Hofmann, R., Alpak, F.O., Dietderich, J., Hunter, S., Day-Stirrat, R.: Effect of image segmentation andamp; voxel size on micro-CT computed effective transport andamp; elastic properties. Mar. Petrol. Geol. 86, 972–990 (2017)

    Article  Google Scholar 

  63. Saxena, N., Hofmann, R., Hows, A., Alpak, F.O., Freeman, J., Hunter, S., Appel, M.: Imaging and computational considerations for digital rock physics. Advances in Water Resources in review (2018)

  64. Saxena, N., Mavko, G., Hofmann, R., Srisutthiyakorn, N.: Estimating permeability from thin sections without reconstruction: graphical study of 3D properties from 2D images. Comput. Geosci. 102, 79–99 (2017)

    Article  Google Scholar 

  65. Skordos, P.A.: Initial and boundary conditions for the lattice Boltzmann method. Phys. Rev. 48, 4823–4842 (1993)

    Article  Google Scholar 

  66. Shah, S.M., Gray, F., Crawshaw, J.P., Boek, E.S.: Micro-computed tomography pore-scale study of flow in porous media: effect of voxel resolution. Adv. Water Res. 95, 276–287 (2016)

    Article  Google Scholar 

  67. Sisavath, S., Jing, X., Pain, C.C., Zimmerman, R.W.: Creeping flow through an axisymmetric sudden contraction or expansion. J. Fluids Eng. 124, 273–278 (2002)

    Article  Google Scholar 

  68. Sisavath, S., Jing, X., Zimmerman, R.W.: Creeping flow through a pipe of varying radius. Phys. Fluids 13, 2762–2772 (2001)

    Article  Google Scholar 

  69. Sochi, T.: Newtonian flow in converging-diverging capillaries. Intern. J. Model. Simul. Scientific Comput. 4, 1350011 (2013)

    Article  Google Scholar 

  70. Succi, S.: The lattice boltzmann equation for fluid dynamics and beyond. numerical mathematics and scientific computation series. Clarendon Press, Oxford (2001)

    Google Scholar 

  71. Sukop, M.C., Thorne, D.T.: Lattice Boltzmann Modeling: An Introduction for Geoscientists and Engineers. Springer-Verlag, Berlin (2006)

    Google Scholar 

  72. Tölke, J.: Implementation of a lattice Boltzmann kernel using the compute unified device architecture developed by nVIDIA. Comput. Visual. Sci. 13(1), 29–39 (2010)

    Article  Google Scholar 

  73. Tölke, J., Baldwin, C., Mu, Y., Derzhi, N., Fang, Q., Grader, A., Dvorkin, J.: Computer simulations of fluid flow in sediment: from images to permeability. Lead. Edge 29, 68–74 (2010)

    Article  Google Scholar 

  74. Torquato, S: Random Heterogeneous Materials: Microstructure and Macroscopic Properties. Springer Science & Business Media, New York (2002)

    Book  Google Scholar 

  75. Wiegmann, A.: Computation of the permeability of porous materials from their microstructure by FFF-Stokes. Berichte des Fraunhofer ITWM, Nr. 129. Kaiserslautern, Germany (2007)

    Google Scholar 

  76. Zhang, D., Zhang, R., Chen, S., Soll, W.E.: Pore scale study of flow in porous media: scale dependency, REV, and statistical REV. Geophys. Res. Lett. 27, 1195–1198 (2000)

    Article  Google Scholar 

Download references

Acknowledgments

We thank Shell International Exploration and Production Inc. for permission to publish this paper. Farrel Gray’s doctoral studentship was provided by the Qatar Carbonates and Carbon Storage Research Centre jointly funded by Qatar Petroleum, Shell and the Qatar Science and Technology Park.

Author information

Authors and Affiliations

  1. Shell International Exploration and Production Inc., 3333 Highway 6 S, Houston, TX, 77082, USA

    F. O. Alpak, N. Saxena, J. Dietderich & R. Hofmann

  2. Department of Computational & Applied Mathematics, Rice University, 6100 Main Street - MS 134, Houston, TX, 77005, USA

    F. O. Alpak

  3. Qatar Carbonates and Carbon Storage Research Centre, Department of Chemical Engineering, Imperial College London, South Kensington Campus, London, SW7 2AZ, England

    F. Gray

  4. Shell Global Solutions International B.V., Badhuisweg 3, 1031, CM, Amsterdam, The Netherlands

    S. Berg

  5. Department of Earth Science & Engineering, Department of Chemical Engineering, Imperial College London, South Kensington Campus, London, SW7 2AZ, England

    S. Berg

Authors
  1. F. O. Alpak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. F. Gray
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. N. Saxena
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. J. Dietderich
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. R. Hofmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. S. Berg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. O. Alpak.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alpak, F.O., Gray, F., Saxena, N. et al. A distributed parallel multiple-relaxation-time lattice Boltzmann method on general-purpose graphics processing units for the rapid and scalable computation of absolute permeability from high-resolution 3D micro-CT images. Comput Geosci 22, 815–832 (2018). https://doi.org/10.1007/s10596-018-9727-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10596-018-9727-7

Keywords

Search

Navigation