Trilinos/Tpetra: Templated Linear Algebra Services Package

Introduction to Tpetra

Tpetra is an attempt at applying generic algorithm techniques to the serial and parallel linear algebra libraries of Epetra. It is hoped that Tpetra will complement Epetra by providing an alternative foundation for future Trilinos solvers. The lessons learned and discoveries made while creating Tpetra are also being applied to improving their Epetra cousins.

Tpetra provides the same functionality as Epetra: the fundamental construction routines and services functions that are required for serial and parallel linear algebra libraries. Unlike Epetra, Tpetra makes extensive use of templates and the Standard Template Library (STL). This provides numerous benefits, including the ability to template the ordinal and scalar fields on any well-defined types, and the use of automatic resource allocation.

Tpetra captures a significant subset of the functionality present in the Epetra package. Functionality not included is either a consequence of different design motivations or of Tpetra's status as a work in progress.

Overview of Tpetra

Templated Types in Tpetra

All of all classes in Tpetra utilize templates, which allows the user to specify any type they want. In some cases, the choice of data type allows increased functionality. For example, 64-bit ordinals allow for problem sizes to break the 2 billion element barrier present in Epetra, whereas complex scalar types allow the native description and solution of complex-valued problems.

Most of the classes in Tpetra are templated according to the data types which constitute the class. These are the following:

• Scalar: A Scalar is the data structure used for storing values. This is the type most likely to be changed by many users. The most common use cases are float, double, complex<float> and complex<double>. However, any data type can be used so long as it implements Teuchos::ScalarTraits and Teuchos::SerializationTraits and supports the necessary arithmetic operations, such as addition, subtraction, division and multiplication.
• LocalOrdinal: A LocalOrdinal is used to store indices representing local IDs. The standard use case, as well as the default for most classes, is int. Any type can be used that implements Teuchos::OrdinalTraits and Teuchos::SerializationTraits and supports the necessary arithmetic. The purpose of template local and global ordinals as two (possibly) different types is for efficiency purposes. For example, the indices of a completed Tpetra::CrsGraph contain objects of type LocalOrdinal. If the specific application allows it, using smaller local ordinals requires less storage and potentially higher rates of computation.
• GlobalOrdinal: A GlobalOrdinal is used to store indices representing global IDs and to describe global properties of a distributed object (e.g., global number of non-zeros in a sparse matrix, global number of rows in a vector.) The GlobalOrdinal therefore dictates the maximum size of a distributed object.

Additionally, computational class in Tpetra will also be templated on a Node type. This node fulfills the Kokkos Node API and allows the Tpetra objects to perform parallel computation on one of a number of shared-memory nodes, including multi-core CPUs and GPUs.

The Tpetra::Distributor class is unique in that it is not parametrized by any templated types. However, the class includes some templated member functions. The Tpetra::Distributor::createFromRecvs() method is templated on the ordinal type used to encode IDs, while Tpetra::Distributor::doPosts() and the other post methods are templated on Packet, the data type being communicated by a particular invocation of the Tpetra::Distributor. This allows a single Tpetra::Distributor object (describing a particular communication pattern) to be used to communicate values of different type.

Tpetra Classes

Tpetra contains a number of classes. The primary parallel classes, employed by most users, are:

• Map class: Tpetra::Map
  Contain information used to distribute vectors, matrices and other objects. This class is analogous to the class Epetra_Map.

• Distributed vector class: Tpetra::MultiVector, Tpetra::Vector
  Provides vector services such as scaling, norms, and dot products.

• Distributed sparse matrix classes: Tpetra::RowMatrix, Tpetra::CrsMatrix
  Tpetra::RowMatrix is a abstract interface for row-distributed sparse matrices. Tpetra::CrsMatrix is a specific implementation of Tpetra::RowMatrix, utilizing compressed row storage format. Both of these classes derive from Tpetra::Operator, the base class for linear operators.

• Block-entry classes: Tpetra::BlockMap, Tpetra::BlockCrsGraph, Tpetra::VbrMatrix, Tpetra::BlockMultiVector

• Import/Export classes: Tpetra::Import and Tpetra::Export
  Allows efficient transfer of objects built using one mapping to a new object with a new mapping. Supports local and global permutations, overlapping Schwarz operations and many other data movement operations.

• Platform classes: Tpetra::HybridPlatform, Tpetra::SerialPlatform, Tpetra::MpiPlatform, Tpetra::DefaultPlatform.
  Platform objects are not required for using Tpetra. However, they provide functionality needed to use other Tpetra classes. Specifically, platform objects allow the easy creation of Teuchos::Comm and Kokkos node objects needed to create Tpetra::Map objects.
  • Tpetra::DefaultPlatform specifies and creates the default platform object (one of the three platform objects below), based on configure time specifications.
  • Tpetra::SerialPlatform and Tpetra::MpiPlatform platform objects return Teuchos::MpiComm and Teuchos::SerialComm objects, along with the default node type as determined at configure time.
  • Tpetra::HybridPlatform is used for building applications where the appropriate node type varies from node to node and is specified by run-time options. This class is helpful when building applications for heterogeneous architectures, such as, e.g., those employing CPUs alongside GPUs.

Trilinos and Tpetra

Tpetra can be used mostly as a stand-alone package, with explicit dependencies on Teuchos and Kokkos. There are adapters allowing the use of Tpetra operators and multivectors in both the Belos linear solver package and the Anasazi eigensolver package.