TNL::Containers::DistributedNDArray< NDArray > Class Template Reference

Distributed N-dimensional array. More...

#include <TNL/Containers/DistributedNDArray.h>

Collaboration diagram for TNL::Containers::DistributedNDArray< NDArray >:

Public Types
using	AllocatorType = typename NDArray::AllocatorType
	Allocator type used for allocating the array.
using	ConstLocalViewType = typename NDArray::ConstViewType
	Compatible constant NDArrayView of the local array.
using	ConstViewType = DistributedNDArrayView< typename NDArray::ConstViewType >
	Compatible constant DistributedNDArrayView type.
using	DeviceType = typename NDArray::DeviceType
	Type of the device used for running operations on the array.
using	IndexType = typename NDArray::IndexType
	Type of indices used for addressing the array elements.
using	LocalBeginsType = LocalBeginsHolder< typename NDArray::SizesHolderType >
	Type which represents the position of the first local element in the global N-dimensional array. It has all static sizes set to 0.
using	LocalRangeType = Subrange< IndexType >
	Type which represents an integer range [a, b).
using	LocalViewType = typename NDArray::ViewType
	Compatible NDArrayView of the local array.
using	OverlapsType = typename NDArray::OverlapsType
	Sequence of integers representing the overlaps in each dimension of a distributed N-dimensional array.
using	PermutationType = typename NDArray::PermutationType
	Permutation that is applied to indices when accessing the array elements.
using	SizesHolderType = typename NDArray::SizesHolderType
	Type of the underlying object which represents the sizes of the N-dimensional array.
using	ValueType = typename NDArray::ValueType
	Type of the values stored in the array.
using	ViewType = DistributedNDArrayView< typename NDArray::ViewType >
	Compatible DistributedNDArrayView type.

Public Member Functions
	DistributedNDArray ()=default
	Constructs an empty array with zero size.
	DistributedNDArray (const AllocatorType &allocator)
	Constructs an empty array and sets the provided allocator.
	DistributedNDArray (const DistributedNDArray &)=default
	Copy constructor (makes a deep copy).
	DistributedNDArray (const DistributedNDArray &other, const AllocatorType &allocator)
	Copy constructor with a specific allocator (makes a deep copy).
	DistributedNDArray (DistributedNDArray &&) noexcept=default
	Move constructor for initialization from rvalues.
void	allocate ()
	Computes the distributed storage size and allocates the local array.
template<typename Device2 = DeviceType, typename Func >
void	forAll (Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all elements of the array.
template<typename Device2 = DeviceType, typename SkipBegins , typename SkipEnds , typename Func >
void	forBoundary (const SkipBegins &skipBegins, const SkipEnds &skipEnds, Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all elements outside the given [skipBegins, skipEnds) range specified by global indices.
template<typename Device2 = DeviceType, typename Func >
void	forBoundary (Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all boundary elements of the array.
template<typename Device2 = DeviceType, typename Func >
void	forGhosts (Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all elements in the ghost region.
template<typename Device2 = DeviceType, typename Begins , typename Ends , typename Func >
void	forInterior (const Begins &begins, const Ends &ends, Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all elements inside the given [begins, ends) range specified by global indices.
template<typename Device2 = DeviceType, typename Func >
void	forInterior (Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all internal elements of the array.
template<typename Device2 = DeviceType, typename Func >
void	forLocalBoundary (Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all local-boundary elements of the array.
template<typename Device2 = DeviceType, typename Func >
void	forLocalInterior (Func f, const typename Device2::LaunchConfiguration &launch_configuration=typename Device2::LaunchConfiguration{}) const
	Evaluates the function f in parallel for all local-internal elements of the array.
AllocatorType	getAllocator () const
	Returns the allocator associated with the array.
const MPI::Comm &	getCommunicator () const
	Returns the MPI communicator associated with the array.
ConstLocalViewType	getConstLocalView () const
	Returns a non-modifiable view of the local array.
ConstViewType	getConstView () const
	Returns a non-modifiable view of the array.
ValueType *	getData ()
	Returns a raw pointer to the local data.
std::add_const_t< ValueType > *	getData () const
	Returns a const-qualified raw pointer to the local data.
template<typename... IndexTypes>
ValueType	getElement (IndexTypes &&... indices) const
	A "safe" accessor for array elements.
LocalBeginsType	getLocalBegins () const
	Returns the beginning position of the local array in the global N-dimensional array.
SizesHolderType	getLocalEnds () const
	Returns the ending position of the local array in the global N-dimensional array.
template<std::size_t level>
LocalRangeType	getLocalRange () const
	Returns a specific [begin, end) subrange of the local array in the global N-dimensional array.
IndexType	getLocalStorageSize () const
	Returns the size (number of elements) needed to store the local N-dimensional array.
LocalViewType	getLocalView ()
	Returns a modifiable view of the local array.
template<std::size_t level>
__cuda_callable__ IndexType	getOverlap () const
	Returns the overlap of a distributed N-dimensional array along the specified axis.
__cuda_callable__ OverlapsType &	getOverlaps ()
	Returns the N-dimensional overlaps holder instance.
__cuda_callable__ const OverlapsType &	getOverlaps () const
	Returns the N-dimensional overlaps holder instance.
template<std::size_t level>
IndexType	getSize () const
	Returns a specific component of the N-dimensional global sizes.
const SizesHolderType &	getSizes () const
	Returns the N-dimensional sizes of the global array.
template<typename... IndexTypes>
IndexType	getStorageIndex (IndexTypes &&... indices) const
	Returns the local storage index for given global indices.
ViewType	getView ()
	Returns a modifiable view of the array.
bool	operator!= (const DistributedNDArray &other) const
	Compares the array with another distributed N-dimensional array.
template<typename... IndexTypes>
ValueType &	operator() (IndexTypes &&... indices)
	Accesses an element of the array.
template<typename... IndexTypes>
const ValueType &	operator() (IndexTypes &&... indices) const
	Accesses an element of the array.
DistributedNDArray &	operator= (const DistributedNDArray &other)=default
	Copy-assignment operator for deep-copying data from another array. Mismatched sizes cause reallocations.
template<typename OtherArray >
DistributedNDArray &	operator= (const OtherArray &other)
	Templated copy-assignment operator for deep-copying data from another array.
DistributedNDArray &	operator= (DistributedNDArray &&) noexcept(false)=default
	Move-assignment operator for acquiring data from rvalues.
bool	operator== (const DistributedNDArray &other) const
	Compares the array with another distributed N-dimensional array.
ValueType &	operator[] (IndexType index)
	Accesses an element in a one-dimensional array.
const ValueType &	operator[] (IndexType index) const
	Accesses an element in a one-dimensional array.
void	reset ()
	Resets the array to the empty state.
template<std::size_t level>
void	setDistribution (IndexType begin, IndexType end, const MPI::Comm &communicator=MPI_COMM_WORLD)
	Sets the distribution of the arrray among MPI ranks along the given axis.
void	setLike (const DistributedNDArray &other)
	Sets sizes of the array to the sizes of an existing array.
template<typename... IndexTypes>
void	setSizes (IndexTypes &&... sizes)
	Sets the global sizes of the array, but does not allocate storage.
void	setValue (ValueType value)
	Sets all elements of the array to given value.

Static Public Member Functions
static constexpr std::size_t	getDimension ()
	Returns the dimension of the N-dimensional array, i.e. N.

Protected Attributes
MPI::Comm	communicator = MPI_COMM_NULL
	MPI communicator associated with the array.
SizesHolderType	globalSizes
	Global sizes of the whole distributed N-dimensional array.
NDArray	localArray
	The N-dimensional array which stores the local elements and overlaps.
LocalBeginsType	localBegins
	Global indices of the first local element in the whole N-dimensional array.
SizesHolderType	localEnds
	Global indices of the end-of-range element, [localBegins, localEnds).

Detailed Description

template<typename NDArray>
class TNL::Containers::DistributedNDArray< NDArray >

Distributed N-dimensional array.

Template Parameters

NDArray

Type of the N-dimensional array which is used to store the local elements. It can be NDArray or SlicedNDArray.

Example:

#include <iostream>
#include <TNL/Containers/BlockPartitioning.h>
#include <TNL/Containers/DistributedNDArray.h>
#include <TNL/Containers/DistributedNDArraySynchronizer.h>
#include <TNL/MPI/ScopedInitializer.h>
 
// The following works for any device (Host, Cuda, etc.)
template< typename Device >
void
distributedNDArrayExample()
{
   using namespace TNL::Containers;
   using LocalArrayType = NDArray< int,                            // Value
                                   SizesHolder< int, 0, 0 >,       // Sizes
                                   std::index_sequence< 1, 0 >,    // Permutation
                                   Device,                         // Device
                                   int,                            // Index
                                   StaticSizesHolder< int, 1, 0 >  // Overlaps
                                   >;
   using ArrayType = DistributedNDArray< LocalArrayType >;
   using LocalRangeType = typename ArrayType::LocalRangeType;
 
   // set input parameters
   const TNL::MPI::Comm communicator = MPI_COMM_WORLD;
   const int num_rows = 10;            // number of rows
   const int num_cols = 4;             // number of columns
   constexpr int distributedAxis = 0;  // 0: num_rows gets distributed, 1: num_cols gets distributed
 
   // decompose the range of rows
   const LocalRangeType localRange =
      TNL::Containers::splitRange< typename LocalArrayType::IndexType >( num_rows, communicator );
 
   // create the distributed array
   ArrayType a;
   a.setSizes( num_rows, num_cols );
   a.template setDistribution< distributedAxis >( localRange.getBegin(), localRange.getEnd(), communicator );
   a.allocate();
 
   // do some work with the array
   auto a_view = a.getLocalView();
   a.forAll(
      [ = ] __cuda_callable__( int gi, int j ) mutable
      {
         // convert global row index to local
         const int i = gi - localRange.getBegin();
         // write the global row index to the array using local coordinates
         a_view( i, j ) = gi;
      } );
   a.forGhosts(
      [ = ] __cuda_callable__( int gi, int j ) mutable
      {
         // convert global row index to local
         const int i = gi - localRange.getBegin();
         // use the local indices to set ghost elements to -1
         a_view( i, j ) = -1;
      } );
 
   // output the local elements as a flat array
   using ArrayView = TNL::Containers::ArrayView< int, Device, int >;
   ArrayView flat_view( a_view.getData(), a_view.getStorageSize() );
   std::cout << "Rank " << communicator.rank() << " before synchronization: " << flat_view << std::endl;
 
   // synchronize the ghost regions and output again
   DistributedNDArraySynchronizer< ArrayType > synchronizer;
   synchronizer.setSynchronizationPattern( NDArraySyncPatterns::D1Q3 );
   synchronizer.synchronize( a );
   std::cout << "Rank " << communicator.rank() << " after synchronization: " << flat_view << std::endl;
}
 
int
main( int argc, char* argv[] )
{
   TNL::MPI::ScopedInitializer mpi( argc, argv );
 
   if( TNL::MPI::GetRank() == 0 )
      std::cout << "The first test runs on CPU ..." << std::endl;
   distributedNDArrayExample< TNL::Devices::Host >();
 
#ifdef __CUDACC__
   TNL::MPI::Barrier();
 
   if( TNL::MPI::GetRank() == 0 )
      std::cout << "The second test runs on GPU ..." << std::endl;
   distributedNDArrayExample< TNL::Devices::Cuda >();
#endif
}

Possible output:

Rank 3: rank on node is 3, using GPU id 1 of 2
Environment:
  CUDA_VISIBLE_DEVICES=
Rank 2: rank on node is 2, using GPU id 0 of 2
Environment:
  CUDA_VISIBLE_DEVICES=
Rank 0: rank on node is 0, using GPU id 0 of 2
Environment:
  CUDA_VISIBLE_DEVICES=
Rank 1: rank on node is 1, using GPU id 1 of 2
Environment:
  CUDA_VISIBLE_DEVICES=
Rank 2 before synchronization: [ -1, 6, 7, -1, -1, 6, 7, -1, -1, 6, 7, -1, -1, 6, 7, -1 ]
Rank 3 before synchronization: [ -1, 8, 9, -1, -1, 8, 9, -1, -1, 8, 9, -1, -1, 8, 9, -1 ]
Rank 1 before synchronization: [ -1, 3, 4, 5, -1, -1, 3, 4, 5, -1, -1, 3, 4, 5, -1, -1, 3, 4, 5, -1 ]
Rank 2 after synchronization: [ 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8 ]
The first test runs on CPU ...
Rank 0 before synchronization: [ -1, 0, 1, 2, -1, -1, 0, 1, 2, -1, -1, 0, 1, 2, -1, -1, 0, 1, 2, -1 ]
Rank 0 after synchronization: [ 9, 0, 1, 2, 3, 9, 0, 1, 2, 3, 9, 0, 1, 2, 3, 9, 0, 1, 2, 3 ]
Rank 1 after synchronization: [ 2, 3, 4, 5, 6, 2, 3, 4, 5, 6, 2, 3, 4, 5, 6, 2, 3, 4, 5, 6 ]
Rank 3 after synchronization: [ 7, 8, 9, 0, 7, 8, 9, 0, 7, 8, 9, 0, 7, 8, 9, 0 ]
The second test runs on GPU ...
Rank 3 before synchronization: [ -1, 8, 9, -1, -1, 8, 9, -1, -1, 8, 9, -1, -1, 8, 9, -1 ]
Rank 2 before synchronization: [ -1, 6, 7, -1, -1, 6, 7, -1, -1, 6, 7, -1, -1, 6, 7, -1 ]
Rank 1 before synchronization: [ -1, 3, 4, 5, -1, -1, 3, 4, 5, -1, -1, 3, 4, 5, -1, -1, 3, 4, 5, -1 ]
Rank 0 before synchronization: [ -1, 0, 1, 2, -1, -1, 0, 1, 2, -1, -1, 0, 1, 2, -1, -1, 0, 1, 2, -1 ]
Rank 2 after synchronization: [ 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8 ]
Rank 0 after synchronization: [ 9, 0, 1, 2, 3, 9, 0, 1, 2, 3, 9, 0, 1, 2, 3, 9, 0, 1, 2, 3 ]
Rank 1 after synchronization: [ 2, 3, 4, 5, 6, 2, 3, 4, 5, 6, 2, 3, 4, 5, 6, 2, 3, 4, 5, 6 ]
Rank 3 after synchronization: [ 7, 8, 9, 0, 7, 8, 9, 0, 7, 8, 9, 0, 7, 8, 9, 0 ]

Member Function Documentation

forAll()

template<typename NDArray >

template<typename Device2 = DeviceType, typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forAll ( Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all elements of the array.

Each MPI rank iterates over all of its local elements.

See NDArrayView::forAll for the requirements on the function f.

forBoundary() [1/2]

template<typename NDArray >

template<typename Device2 = DeviceType, typename SkipBegins , typename SkipEnds , typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forBoundary ( const SkipBegins & skipBegins, const SkipEnds & skipEnds, Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all elements outside the given [skipBegins, skipEnds) range specified by global indices.

Each MPI rank iterates over its local elements outside the range.

See NDArrayView::forAll for the requirements on the function f.

forBoundary() [2/2]

template<typename NDArray >

template<typename Device2 = DeviceType, typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forBoundary ( Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all boundary elements of the array.

Each MPI rank iterates over its local elements which are neighbours of global boundaries.

See NDArrayView::forAll for the requirements on the function f.

forGhosts()

template<typename NDArray >

template<typename Device2 = DeviceType, typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forGhosts ( Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all elements in the ghost region.

Each MPI rank iterates over elements which are in the overlapping region (i.e., owned by a different MPI rank). If all overlaps are 0, it has no effect.

See NDArrayView::forAll for the requirements on the function f.

forInterior() [1/2]

template<typename NDArray >

template<typename Device2 = DeviceType, typename Begins , typename Ends , typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forInterior ( const Begins & begins, const Ends & ends, Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all elements inside the given [begins, ends) range specified by global indices.

Each MPI rank iterates over its local elements from the range.

See NDArrayView::forAll for the requirements on the function f.

forInterior() [2/2]

template<typename NDArray >

template<typename Device2 = DeviceType, typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forInterior ( Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all internal elements of the array.

Each MPI rank iterates over its local elements which are not neighbours of global boundaries.

See NDArrayView::forAll for the requirements on the function f.

forLocalBoundary()

template<typename NDArray >

template<typename Device2 = DeviceType, typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forLocalBoundary ( Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all local-boundary elements of the array.

Each MPI rank iterates over its local elements which are neighbours of overlaps. If all overlaps are 0, it has no effect.

See NDArrayView::forAll for the requirements on the function f.

forLocalInterior()

template<typename NDArray >

template<typename Device2 = DeviceType, typename Func >

void TNL::Containers::DistributedNDArray< NDArray >::forLocalInterior ( Func f, const typename Device2::LaunchConfiguration & launch_configuration = typename Device2::LaunchConfiguration{} ) const

inline

Evaluates the function f in parallel for all local-internal elements of the array.

Each MPI rank iterates over its local elements which are not neighbours of overlaps. If all overlaps are 0, it is equivalent to forAll.

See NDArrayView::forAll for the requirements on the function f.

getElement()

template<typename NDArray >

template<typename... IndexTypes>

ValueType TNL::Containers::DistributedNDArray< NDArray >::getElement ( IndexTypes &&... indices ) const

inlinenodiscard

A "safe" accessor for array elements.

It can be called only from the host code and it will do a slow copy from the device.

getLocalRange()

template<typename NDArray >

template<std::size_t level>

LocalRangeType TNL::Containers::DistributedNDArray< NDArray >::getLocalRange ( ) const

inlinenodiscard

Returns a specific [begin, end) subrange of the local array in the global N-dimensional array.

Template Parameters

level

Integer specifying the component of the sizes to be returned.

getOverlap()

template<typename NDArray >

template<std::size_t level>

__cuda_callable__ IndexType TNL::Containers::DistributedNDArray< NDArray >::getOverlap ( ) const

inlinenodiscard

Returns the overlap of a distributed N-dimensional array along the specified axis.

Template Parameters

level

Integer specifying the axis of the array.

getSize()

template<typename NDArray >

template<std::size_t level>

IndexType TNL::Containers::DistributedNDArray< NDArray >::getSize ( ) const

inlinenodiscard

Returns a specific component of the N-dimensional global sizes.

Template Parameters

level

Integer specifying the component of the sizes to be returned.

getStorageIndex()

template<typename NDArray >

template<typename... IndexTypes>

IndexType TNL::Containers::DistributedNDArray< NDArray >::getStorageIndex ( IndexTypes &&... indices ) const

inlinenodiscard

Returns the local storage index for given global indices.

Parameters

indices

Global indices of the element in the N-dimensional array. The number of indices supplied must be equal to N, i.e. getDimension().

Returns

An index that can be used to address the element in a local one-dimensional array.

operator()() [1/2]

template<typename NDArray >

template<typename... IndexTypes>

ValueType & TNL::Containers::DistributedNDArray< NDArray >::operator() ( IndexTypes &&... indices )

inlinenodiscard

Accesses an element of the array.

Only local elements or elements in the overlapping region can be accessed.

Parameters

indices

Global indices of the element in the N-dimensional array. The number of indices supplied must be equal to N, i.e. getDimension().

Returns

Reference to the array element.

operator()() [2/2]

template<typename NDArray >

template<typename... IndexTypes>

const ValueType & TNL::Containers::DistributedNDArray< NDArray >::operator() ( IndexTypes &&... indices ) const

inlinenodiscard

Accesses an element of the array.

Only local elements or elements in the overlapping region can be accessed.

Parameters

indices

Global indices of the element in the N-dimensional array. The number of indices supplied must be equal to N, i.e. getDimension().

Returns

Constant reference to the array element.

operator[]() [1/2]

template<typename NDArray >

ValueType & TNL::Containers::DistributedNDArray< NDArray >::operator[] ( IndexType index )

inlinenodiscard

Accesses an element in a one-dimensional array.

Only local elements or elements in the overlapping region can be accessed.

Warning

This function can be used only when the dimension of the array is equal to 1.

Parameters

index

Global index of the element in the one-dimensional array.

Returns

Reference to the array element.

operator[]() [2/2]

template<typename NDArray >

const ValueType & TNL::Containers::DistributedNDArray< NDArray >::operator[] ( IndexType index ) const

inlinenodiscard

Accesses an element in a one-dimensional array.

Only local elements or elements in the overlapping region can be accessed.

Warning

This function can be used only when the dimension of the array is equal to 1.

Parameters

index

Global index of the element in the one-dimensional array.

Returns

Reference to the array element.

reset()

template<typename NDArray >

void TNL::Containers::DistributedNDArray< NDArray >::reset ( )

inline

Resets the array to the empty state.

The current data will be deallocated, thus all pointers and views to the array elements will become invalid.

setDistribution()

template<typename NDArray >

template<std::size_t level>

void TNL::Containers::DistributedNDArray< NDArray >::setDistribution ( IndexType begin, IndexType end, const MPI::Comm & communicator = MPI_COMM_WORLD )

inline

Sets the distribution of the arrray among MPI ranks along the given axis.

Template Parameters

level

Integer specifying the axis of the array to be distributed.

Parameters

begin	Index of the first element owned by the calling MPI rank.
end	Index of the first element not owned by the calling MPI rank. I.e., the calling rank owns the range [begin, end).
communicator	MPI communicator to associate with the array.

setLike()

template<typename NDArray >

void TNL::Containers::DistributedNDArray< NDArray >::setLike ( const DistributedNDArray< NDArray > & other )

inline

Sets sizes of the array to the sizes of an existing array.

If the array size changes, the current data will be deallocated, thus all pointers and views to the array alements will become invalid.

Member Data Documentation

localBegins

template<typename NDArray >

LocalBeginsType TNL::Containers::DistributedNDArray< NDArray >::localBegins

protected

Global indices of the first local element in the whole N-dimensional array.

Note that localBegins and localEnds have different static sizes (and hence different C++ type): localBegins is always 0, localEnds has always the full static size.

---

The documentation for this class was generated from the following file:

• src/TNL/Containers/DistributedNDArray.h