Skip to content

Error from benchmark/test distributed nonhydrostatic model scripts #2347

New issue
New issue
Closed
#2502
Closed
Error from benchmark/test distributed nonhydrostatic model scripts#2347
#2502
Labels
bug 🐞Even a perfect program still has bugsdistributed 🕸️Our plan for total cluster domination
@johnryantaylor

Description

@johnryantaylor
johnryantaylor
opened on Mar 13, 2022

I am getting the following error from benchmark/distributed_nonhydrostatic_model.jl where I have set threaded = false and decomposition = Slab(). I get a very similar error from the distributed nonhydrostatic script when I run the tests script, but the distributed shallow water model runs fine. In case it matters, I'm running mpich/3.0.4 (a bit old) on a linux system running Ubuntu.

Oceananigans v0.71.5
Julia Version 1.6.5
Commit 9058264a69 (2021-12-19 12:30 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, westmere)
[2022/03/13 19:01:45.515] INFO  Benchmarking weak scaling nonhydrostatic model with Slab decomposition [N=(128, 128, 16), ranks=(1, 1, 1)]...
[2022/03/13 19:02:09.310] INFO  Setting up distributed nonhydrostatic model with N=(128, 128, 16) grid points and ranks=(1, 1, 1) on rank 0...
[2022/03/13 19:02:52.561] INFO  Warming up distributed nonhydrostatic model on rank 0...
[2022/03/13 19:04:45.955] INFO  Benchmarking distributed nonhydrostatic model on rank 0...
[2022/03/13 19:04:50.814] INFO  Done benchmarking on rank 0. Median time: 72.806 ms
[2022/03/13 19:04:57.400] INFO  Benchmarking weak scaling nonhydrostatic model with Slab decomposition [N=(128, 128, 32), ranks=(1, 2, 1)]...
[2022/03/13 19:05:21.386] INFO  Setting up distributed nonhydrostatic model with N=(128, 128, 32) grid points and ranks=(1, 2, 1) on rank 1...
[2022/03/13 19:05:21.430] INFO  Setting up distributed nonhydrostatic model with N=(128, 128, 32) grid points and ranks=(1, 2, 1) on rank 0...
[2022/03/13 19:06:04.003] INFO  Warming up distributed nonhydrostatic model on rank 1...
[2022/03/13 19:06:04.004] INFO  Warming up distributed nonhydrostatic model on rank 0...
ERROR: ERROR: LoadError: LoadError: DimensionMismatch(DimensionMismatch("arrays could not be broadcast to a common size; got a dimension with lengths 128 and 64")
Stacktrace:
  [1] "arrays could not be broadcast to a common size; got a dimension with lengths 128 and 64")
Stacktrace:
  [1] _bcs1
    @ ./broadcast.jl:501 [inlined]
  [2] _bcs1
    @ ./broadcast.jl:501 [inlined]
  [2] _bcs(_bcs(shape::shape::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}, newshape::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}})
    @ Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}, newshape::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}})
    @ Base.Broadcast ./broadcast.jl:495
  [3] broadcast_shape
    @ ./broadcast.jl:489 [inlined]
  [4] combine_axes
    @ ./broadcast.jl:484 [inlined]
  [5] _axes
    @ ./broadcast.jl:209 [inlined]
  [6] axes
    @ ./broadcast.jl:207 [inlined]
  [7] _unwrap_pa(bc::Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(/), Tuple{Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(-), Tuple{PencilArrays.PencilArrayBroadcastable{ComplexF64, 3, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}}}}, Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{3}, Nothing, typeof(+), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}}})
    @ PencilArrays ~/.julia/packages/PencilArrays/Jy1Lp/src/broadcast.jl:51
  [8] materialize!(u::PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}, bc_in::Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(/), Tuple{Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(-), Tuple{PencilArrays.PencilArrayBroadcastable{ComplexF64, 3, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}}}}, Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{3}, Nothing, typeof(+), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}}})
    @ PencilArrays ~/.julia/packages/PencilArrays/Jy1Lp/src/broadcast.jl:33
  [9] solve!(x::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}, solver::Base.Broadcast ./broadcast.jl:495
  [3] broadcast_shape
    @ ./broadcast.jl:489 [inlined]
  [4] combine_axes
    @ ./broadcast.jl:484 [inlined]
  [5] _axes
    @ ./broadcast.jl:209 [inlined]
  [6] axes
    @ ./broadcast.jl:207 [inlined]
  [7] _unwrap_pa(bc::Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(/), Tuple{Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(-), Tuple{PencilArrays.PencilArrayBroadcastable{ComplexF64, 3, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}}}}, Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{3}, Nothing, typeof(+), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}}})
    @ PencilArrays ~/.julia/packages/PencilArrays/Jy1Lp/src/broadcast.jl:51
  [8] materialize!(u::PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}, bc_in::DistributedFFTBasedPoissonSolver{PencilFFTs.PencilFFTPlan{ComplexF64, 3, true, 3, 2, 0, PencilFFTs.GlobalFFTParams{Float64, 3, true, Tuple{PencilFFTs.Transforms.FFT!, PencilFFTs.Transforms.FFT!, PencilFFTs.Transforms.FFT!}}, Tuple{PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}, PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}, PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}}, PencilArrays.Transpositions.PointToPoint}, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CPU}, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, NamedTuple{(:λx, :λy, :λz), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}, PencilArrays.ManyPencilArray{ComplexF64, 3, 3, Tuple{PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}}, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}}}, Vector{ComplexF64}}})
    @ Oceananigans.Distributed /mhome/damtp/atmos/jrt51/Oceananigans.jl/src/Distributed/distributed_fft_based_poisson_solver.jl:65
 [10] solve_for_pressure!(pressure::Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(/), Tuple{Base.Broadcast.Broadcasted{PencilArrays.PencilArrayStyle{3}, Nothing, typeof(-), Tuple{PencilArrays.PencilArrayBroadcastable{ComplexF64, 3, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}}}}, Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{3}, Nothing, typeof(+), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}}})
    @ PencilArrays ~/.julia/packages/PencilArrays/Jy1Lp/src/broadcast.jl:33
  [9] solve!(x::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}, solver::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}, solver::DistributedFFTBasedPoissonSolver{PencilFFTs.PencilFFTPlan{ComplexF64, 3, true, 3, 2, 0, PencilFFTs.GlobalFFTParams{Float64, 3, true, Tuple{PencilFFTs.Transforms.FFT!, PencilFFTs.Transforms.FFT!, PencilFFTs.Transforms.FFT!}}, Tuple{PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}, PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}, PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}}, PencilArrays.Transpositions.PointToPoint}, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CPU}, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, NamedTuple{(:λx, :λy, :λz), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}, PencilArrays.ManyPencilArray{ComplexF64, 3, 3, Tuple{PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}}, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}}}, Vector{ComplexF64}}}, Δt::Int64, U★::DistributedFFTBasedPoissonSolver{PencilFFTs.PencilFFTPlan{ComplexF64, 3, true, 3, 2, 0, PencilFFTs.GlobalFFTParams{Float64, 3, true, Tuple{PencilFFTs.Transforms.FFT!, PencilFFTs.Transforms.FFT!, PencilFFTs.Transforms.FFT!}}, Tuple{PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}, PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}, PencilFFTs.PencilPlan1D{ComplexF64, ComplexF64, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}, PencilFFTs.Transforms.FFT!, FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Int64}, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Int64}}}, PencilArrays.Transpositions.PointToPoint}, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, CPU}, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, NamedTuple{(:λx, :λy, :λz), Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}}, PencilArrays.ManyPencilArray{ComplexF64, 3, 3, Tuple{PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.NoPermutation, Vector{UInt8}}}, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(2, 1, 3), 3}, Vector{UInt8}}}, PencilArrays.PencilArray{ComplexF64, 3, Base.ReshapedArray{ComplexF64, 3, SubArray{ComplexF64, 1, Vector{ComplexF64}, Tuple{Base.OneTo{Int64}}, true}, Tuple{}}, 3, 0, PencilArrays.Pencils.Pencil{3, 2, StaticPermutations.Permutation{(3, 2, 1), 3}, Vector{UInt8}}}}, Vector{ComplexF64}}})
    @ Oceananigans.Distributed /mhome/damtp/atmos/jrt51/Oceananigans.jl/src/Distributed/distributed_fft_based_poisson_solver.jl:65
 [10] solve_for_pressure!(pressure::Field{Center, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}, solver::NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float64, Periodic, Periodic, Periodic, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}, MultiArch{CPU, Int64, Tuple{Int64, Int64, Int64}, Tuple{Int64, Int64, Int64}, Oceananigans.Distributed.RankConnectivity{Nothing, Nothing, Int64, Int64, Nothing, Nothing}, MPI.Comm}}, Float64, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{HaloCommunication, Oceananigans.Distributed.HaloCommunicationRanks{Int64, Int64}}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing}}})
    @ Oceananigans.Models.NonhydrostaticModels /mhome/damtp/atmos/jrt51/Oceananigans.jl/src/Models/NonhydrostaticModels/solve_for_pressure.jl:35
 [11] calculate_pressure_correction!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug 🐞Even a perfect program still has bugsdistributed 🕸️Our plan for total cluster domination

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions