|
Tpetra Matrix/Vector Services
Version of the Day
|
Base class for distributed Tpetra objects that support data redistribution. More...
#include <Tpetra_DistObject.hpp>

Public Member Functions | |
Constructors and destructor | |
| DistObject (const Teuchos::RCP< const Map< LocalOrdinal, GlobalOrdinal, Node > > &map) | |
| Constructor. | |
| DistObject (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source) | |
| Copy constructor. | |
| virtual | ~DistObject () |
| Destructor (virtual for memory safety of derived classes). | |
Public methods for redistributing data | |
| void | doImport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, const Import< LocalOrdinal, GlobalOrdinal, Node > &importer, CombineMode CM) |
| Import using an Import object ("forward mode"). | |
| void | doExport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &dest, const Export< LocalOrdinal, GlobalOrdinal, Node > &exporter, CombineMode CM) |
| Export using an Export object ("forward mode"). | |
| void | doImport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, const Export< LocalOrdinal, GlobalOrdinal, Node > &exporter, CombineMode CM) |
| Import using an Export object ("reverse mode"). | |
| void | doExport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &dest, const Import< LocalOrdinal, GlobalOrdinal, Node > &importer, CombineMode CM) |
| Export using an Import object ("reverse mode"). | |
Attribute accessor methods | |
| bool | isDistributed () const |
| Whether this is a globally distributed object. | |
| const Teuchos::RCP< const Map < LocalOrdinal, GlobalOrdinal, Node > > & | getMap () const |
| The Map with which this DistObject was constructed. | |
I/O methods | |
| void | print (std::ostream &os) const |
| Print this object to the given output stream. | |
Implementation of \c Teuchos::Describable | |
| virtual std::string | description () const |
| One-line descriptiion of this object. | |
| virtual void | describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const |
| Print a descriptiion of this object to the given output stream. | |
Protected Types | |
| enum | ReverseOption |
| Whether the data transfer should be performed in forward or reverse mode. More... | |
Protected Member Functions | |
| virtual void | doTransfer (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, CombineMode CM, size_t numSameIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteToLIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteFromLIDs, const Teuchos::ArrayView< const LocalOrdinal > &remoteLIDs, const Teuchos::ArrayView< const LocalOrdinal > &exportLIDs, Distributor &distor, ReverseOption revOp) |
| Redistribute data across memory images. | |
| virtual void | createViews () const |
| Hook for creating a const view. | |
| virtual void | createViewsNonConst (Kokkos::ReadWriteOption rwo) |
| Hook for creating a nonconst view. | |
| virtual void | releaseViews () const |
| Hook for releasing views. | |
Methods implemented by subclasses and used by \c doTransfer(). | |
The | |
| virtual bool | checkSizes (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source)=0 |
| Compare the source and target (this) objects for compatibility. | |
| virtual void | copyAndPermute (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, size_t numSameIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteToLIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteFromLIDs)=0 |
| Perform copies and permutations that are local to this process. | |
| virtual void | packAndPrepare (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, const Teuchos::ArrayView< const LocalOrdinal > &exportLIDs, Teuchos::Array< Packet > &exports, const Teuchos::ArrayView< size_t > &numPacketsPerLID, size_t &constantNumPackets, Distributor &distor)=0 |
| Perform any packing or preparation required for communication. | |
| virtual void | unpackAndCombine (const Teuchos::ArrayView< const LocalOrdinal > &importLIDs, const Teuchos::ArrayView< const Packet > &imports, const Teuchos::ArrayView< size_t > &numPacketsPerLID, size_t constantNumPackets, Distributor &distor, CombineMode CM)=0 |
| Perform any unpacking and combining after communication. | |
Protected Attributes | |
| Teuchos::RCP< const Map < LocalOrdinal, GlobalOrdinal, Node > > | map_ |
| The Map over which this object is distributed. | |
Base class for distributed Tpetra objects that support data redistribution.
DistObject is a base class for all Tpetra distributed global objects, including CrsMatrix and MultiVector. It provides the basic mechanisms and interface specifications for importing and exporting operations using Import and Export objects.
| LocalOrdinal | The type of local IDs. Same as Map's LocalOrdinal template parameter. This should be an integer type, preferably signed. |
| GlobalOrdinal | The type of global IDs. Same as Map's GlobalOrdinal template parameter. Defaults to the same type as LocalOrdinal. This should also be an integer type, preferably signed. |
| Node | Same as Map's Node template parameter. Defaults to the default Kokkos Node type. |
Most Tpetra users will create subclasses of DistObject (like CrsMatrix or MultiVector), describe data distribution via Map instances, create data redistribution plans via Import or Export, and invoke the doImport() or doExport() methods of DistObject with an Import or Export object to redistribute data. Thus, the only methods of DistObject of interest to most Tpetra users are doImport() and doExport().
If you want to implement your own DistObject subclass, you should start by implementing the four pure virtual methods: checkSizes(), copyAndPermute(), packAndPrepare(), and unpackAndCombine(). The implementation of doTransfer() includes documentation that explains how DistObject uses those methods to do data redistribution.
If you are writing a DistObject class that uses Kokkos compute buffers and aims to work for any Kokkos Node type, you should also implement the three hooks that create and release views: createViews(), createViewsNonConst(), and releaseViews(). The default implementation of these hooks does nothing. The documentation of these methods explains different ways you might choose to implement them.
Distributed Tpetra objects may be either "distributed global" or "replicated local." Distributed global objects are partitioned across multiple processes in a communicator. Each process owns at least one element in the object's Map that is not owned by another process. For replicated local objects, each element in the object's Map is owned redundantly by all processes in the object's communicator. Some algorithms use objects that are too small to be distributed across all processes. The upper Hessenberg matrix in a GMRES iterative solve is a good example. In other cases, such as with block iterative methods, block dot product functions produce small dense matrices that are required by all images. Replicated local objects handle these situations.
Definition at line 117 of file Tpetra_DistObject.hpp.
enum Tpetra::DistObject::ReverseOption [protected] |
Whether the data transfer should be performed in forward or reverse mode.
"Reverse mode" means calling doExport() with an Import object, or calling doImport() with an Export object. "Forward mode" means calling doExport() with an Export object, or calling doImport() with an Import object.
Definition at line 216 of file Tpetra_DistObject.hpp.
| Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject | ( | const Teuchos::RCP< const Map< LocalOrdinal, GlobalOrdinal, Node > > & | map | ) | [explicit] |
Constructor.
Definition at line 426 of file Tpetra_DistObject.hpp.
| Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source | ) |
Copy constructor.
Definition at line 432 of file Tpetra_DistObject.hpp.
| Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::~DistObject | ( | ) | [virtual] |
Destructor (virtual for memory safety of derived classes).
Definition at line 437 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doImport | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source, |
| const Import< LocalOrdinal, GlobalOrdinal, Node > & | importer, | ||
| CombineMode | CM | ||
| ) |
Import using an Import object ("forward mode").
Definition at line 481 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doExport | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | dest, |
| const Export< LocalOrdinal, GlobalOrdinal, Node > & | exporter, | ||
| CombineMode | CM | ||
| ) |
Export using an Export object ("forward mode").
Definition at line 506 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doImport | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source, |
| const Export< LocalOrdinal, GlobalOrdinal, Node > & | exporter, | ||
| CombineMode | CM | ||
| ) |
Import using an Export object ("reverse mode").
Definition at line 528 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doExport | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | dest, |
| const Import< LocalOrdinal, GlobalOrdinal, Node > & | importer, | ||
| CombineMode | CM | ||
| ) |
Export using an Import object ("reverse mode").
Definition at line 550 of file Tpetra_DistObject.hpp.
| bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::isDistributed | ( | ) | const [inline] |
Whether this is a globally distributed object.
For a definition of "globally distributed" (and its opposite, "locally replicated"), see the documentation of Map's isDistributed() method.
Definition at line 573 of file Tpetra_DistObject.hpp.
| const Teuchos::RCP<const Map<LocalOrdinal,GlobalOrdinal,Node> >& Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::getMap | ( | ) | const [inline] |
The Map with which this DistObject was constructed.
Definition at line 172 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::print | ( | std::ostream & | os | ) | const |
Print this object to the given output stream.
We generally assume that all MPI processes can print to the given stream.
Definition at line 719 of file Tpetra_DistObject.hpp.
| std::string Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::description | ( | ) | const [virtual] |
One-line descriptiion of this object.
We declare this method virtual so that subclasses of DistObject may override it.
Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, and Tpetra::Vector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.
Definition at line 442 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::describe | ( | Teuchos::FancyOStream & | out, |
| const Teuchos::EVerbosityLevel | verbLevel = Teuchos::Describable::verbLevel_default |
||
| ) | const [virtual] |
Print a descriptiion of this object to the given output stream.
We declare this method virtual so that subclasses of DistObject may override it.
Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, and Tpetra::Vector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.
Definition at line 459 of file Tpetra_DistObject.hpp.
| void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doTransfer | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source, |
| CombineMode | CM, | ||
| size_t | numSameIDs, | ||
| const Teuchos::ArrayView< const LocalOrdinal > & | permuteToLIDs, | ||
| const Teuchos::ArrayView< const LocalOrdinal > & | permuteFromLIDs, | ||
| const Teuchos::ArrayView< const LocalOrdinal > & | remoteLIDs, | ||
| const Teuchos::ArrayView< const LocalOrdinal > & | exportLIDs, | ||
| Distributor & | distor, | ||
| ReverseOption | revOp | ||
| ) | [protected, virtual] |
Redistribute data across memory images.
| source | [in] The source object, to redistribute into the destination object, which is *this object. |
| CM | [in] The combine mode that describes how to combine values that map to the same global ID on the same process. |
| permuteToLIDs | [in] See copyAndPermute(). |
| permuteFromLIDs | [in] See copyAndPermute(). |
| remoteLIDs | [in] List of entries (as local IDs) in the destination object to receive from other processes. |
| exportLIDs | [in] See packAndPrepare(). |
| distor | [in/out] The Distributor object that knows how to redistribute data. |
| revOp | [in] Whether to do a forward or reverse mode redistribution. |
Definition at line 580 of file Tpetra_DistObject.hpp.
| virtual bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::checkSizes | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source | ) | [protected, pure virtual] |
Compare the source and target (this) objects for compatibility.
Implemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, and Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >.
| virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::copyAndPermute | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source, |
| size_t | numSameIDs, | ||
| const Teuchos::ArrayView< const LocalOrdinal > & | permuteToLIDs, | ||
| const Teuchos::ArrayView< const LocalOrdinal > & | permuteFromLIDs | ||
| ) | [protected, pure virtual] |
Perform copies and permutations that are local to this process.
| source | [in] On entry, the source object, from which we are distributing. We distribute to the destination object, which is *this object. |
| numSameIDs | [in] The umber of elements that are the same on the source and destination (this) objects. These elements are owned by the same process in both the source and destination objects. No permutation occurs. |
| numPermuteIDs | [in] The number of elements that are locally permuted between the source and destination objects. |
| permuteToLIDs | [in] List of the elements that are permuted. They are listed by their LID in the destination object. |
| permuteFromLIDs | [in] List of the elements that are permuted. They are listed by their LID in the source object. |
Implemented in Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >.
| virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::packAndPrepare | ( | const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & | source, |
| const Teuchos::ArrayView< const LocalOrdinal > & | exportLIDs, | ||
| Teuchos::Array< Packet > & | exports, | ||
| const Teuchos::ArrayView< size_t > & | numPacketsPerLID, | ||
| size_t & | constantNumPackets, | ||
| Distributor & | distor | ||
| ) | [protected, pure virtual] |
Perform any packing or preparation required for communication.
| source | [in] Source object for the redistribution. |
| exportLIDs | [in] List of the entries (as local IDs in the source object) we will be sending to other images. |
| exports | [out] On exit, the buffer for data to send. |
| numPacketsPerLID | [out] On exit, numPacketsPerLID[i] contains the number of packets to be exported for exportLIDs[i]. If constantNumPackets is nonzero, you should use that instead, and not rely on numPacketsPerLID[i] being filled. |
| constantNumPackets | [out] On exit, 0 if numPacketsPerLID has variable contents (different size for each LID). If nonzero, then it is expected that num-packets-per-LID is constant, and constantNumPackets holds that value. |
| distor | [in] The Distributor object we are using. |
Implemented in Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >.
| virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::unpackAndCombine | ( | const Teuchos::ArrayView< const LocalOrdinal > & | importLIDs, |
| const Teuchos::ArrayView< const Packet > & | imports, | ||
| const Teuchos::ArrayView< size_t > & | numPacketsPerLID, | ||
| size_t | constantNumPackets, | ||
| Distributor & | distor, | ||
| CombineMode | CM | ||
| ) | [protected, pure virtual] |
Perform any unpacking and combining after communication.
| importLIDs | [in] List of the entries (as LIDs in the destination object) we received from other images. |
| imports | [in] Buffer containing data we received. |
| numPacketsPerLID | [in] numPacketsPerLID[i] contains the number of packets imported for importLIDs[i]. |
| constantNumPackets | [in] If nonzero, then numPacketsPerLID is constant (same value in all entries) and constantNumPackets is that value. If zero, use numPacketsPerLID[i] instead. |
| distor | [in] The Distributor object we are using. |
| CM | [in] The combine mode to use when combining the imported entries with existing entries. |
Implemented in Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, and Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >.
| virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::createViews | ( | ) | const [inline, protected, virtual] |
Hook for creating a const view.
doTransfer() calls this on the source object. By default, it does nothing, but the source object can use this as a hint to fetch data from a compute buffer on an off-CPU device (such as a GPU) into host memory.
Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.
Definition at line 358 of file Tpetra_DistObject.hpp.
| virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::createViewsNonConst | ( | Kokkos::ReadWriteOption | rwo | ) | [inline, protected, virtual] |
Hook for creating a nonconst view.
doTransfer() calls this on the destination (*this) object. By default, it does nothing, but the destination object can use this as a hint to fetch data from a compute buffer on an off-CPU device (such as a GPU) into host memory.
| rwo | [in] Whether to create a write-only or a read-and-write view. For Kokkos Node types where compute buffers live in a separate memory space (e.g., in the device memory of a discrete accelerator like a GPU), a write-only view only requires copying from host memory to the compute buffer, whereas a read-and-write view requires copying both ways (once to read, from the compute buffer to host memory, and once to write, back to the compute buffer). |
Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.
Definition at line 375 of file Tpetra_DistObject.hpp.
| virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::releaseViews | ( | ) | const [inline, protected, virtual] |
Hook for releasing views.
doTransfer() calls this on both the source and destination objects, once it no longer needs to access that object's data. By default, this method does nothing. Implementations may use this as a hint to free host memory which is a view of a compute buffer, once the host memory view is no longer needed. Some implementations may prefer to mirror compute buffers in host memory; for these implementations, releaseViews() may do nothing.
Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.
Definition at line 387 of file Tpetra_DistObject.hpp.
Teuchos::RCP<const Map<LocalOrdinal,GlobalOrdinal,Node> > Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::map_ [protected] |
The Map over which this object is distributed.
Definition at line 390 of file Tpetra_DistObject.hpp.
1.7.6.1