Collaboration diagram for Sierra Toolkit Rebalance Unit Tests:

Simple Basic Unit Test

This unit test creates a 2D quad mesh on proc 0 and then moves these elements in a predetermined way to other procs and finally all to proc 1 in parallel. The test passes in serial trivially and in paralle if all entities end up on proc 1.

The relevant headers for this unit test are

// relevant headers
#include <stk_rebalance/Rebalance.hpp>
#include <stk_rebalance/Partition.hpp>
#include <stk_rebalance_utils/RebalanceUtils.hpp>
// end relevant headers

The following quad4 mesh :

Global node and element numbering

           7       8       9
           +-------+-------+
           |       |       |
           |  e3   |  e4   |
           |       |5      |
          4+-------+-------+6   
           |       |       |     Y
           |  e1   |  e2   |     |
           |       |       |     |
           +-------+-------+     *--> X
           1       2      3

Local node numbering

is constructed on proc 0 by the code

  // create initial mesh
  bulk_data.modification_begin();

  if ( p_rank == 0 ) {
    const unsigned nnx = nx + 1 ;
    for ( unsigned iy = 0 ; iy < ny ; ++iy ) {
      for ( unsigned ix = 0 ; ix < nx ; ++ix ) {
        stk_classic::mesh::EntityId elem = 1 + ix + iy * nx ;
        stk_classic::mesh::EntityId nodes[4] ;
        nodes[0] = 1 + ix + iy * nnx ;
        nodes[1] = 2 + ix + iy * nnx ;
        nodes[2] = 2 + ix + ( iy + 1 ) * nnx ;
        nodes[3] = 1 + ix + ( iy + 1 ) * nnx ;

        stk_classic::mesh::fem::declare_element( bulk_data , quad_part , elem , nodes );
      }
    }
    // end create initial mesh

Uniform unit weights are assigned to elements via

    // assign element weights
    for ( unsigned iy = 0 ; iy < ny ; ++iy ) {
      for ( unsigned ix = 0 ; ix < nx ; ++ix ) {
        stk_classic::mesh::EntityId elem = 1 + ix + iy * nx ;
        stk_classic::mesh::Entity * e = bulk_data.get_entity( element_rank, elem );
        double * const e_weight = stk_classic::mesh::field_data( weight_field , *e );
        *e_weight = 1.0;
      }
    }
    // end assign element weights

Mesh creation is signaled complete by calling

  bulk_data.modification_end();

We instantiate our MockPartition, our Selector to include all elements in the mesh and our imbalance threshold we want to use to determine whether or not to compute a new element partition among processors:

  // Create our Partition and Selector objects
  MockPartition partition(fem_meta, bulk_data);
  stk_classic::mesh::Selector selector(fem_meta.universal_part());
/// \skipline double imblance_threshhold

We interrogate whether a rebalance is needed by calling

/// \skipline rebalance::check_balance

If the current imbalance is above the threshold, the call returns true and we can then invoke the actual rebalance via

/// \until rebalance::rebalance

Upon return the bulk_data will reflect the new partiioning of elements with dependent entities, eg faces, edges, nodes, partitioned in a default greedy manner.

See UnitTestSimple.cpp for the complete source listing.

Simple Zoltan Unit Test using Graph-Based Partitioning

This unit test is identical to Simple Zoltan Unit Test with the only difference being use of graph-based partitioning instead of element weights.

The distinction involves setting up an appropriate parameter list,

  // Configure Zoltan to use graph-based partitioning
  Teuchos::ParameterList graph;
  Teuchos::ParameterList lb_method;
  lb_method.set("LOAD BALANCING METHOD"      , "4");
  graph.sublist(stk_classic::rebalance::Zoltan::default_parameters_name())=lb_method;
  stk_classic::rebalance::Zoltan zoltan_partition(comm, spatial_dimension, graph);
  // end configure snippet

and then calling check_balance and rebalance with NULL weight fields and calling rebalance with non-NULL coordinates, eg

  const double imbalance_threshold = stk_classic::rebalance::check_balance(bulk_data, NULL, fem_meta.node_rank(), &selector);
  const bool do_rebal = 1.5 < imbalance_threshold;

  // Check that we satisfy our threshhold
  STKUNIT_ASSERT( !do_rebal );
  if( (2 == p_size) || (4 == p_size) )
  {
    STKUNIT_ASSERT_NEAR(imbalance_threshold, 1.0, .1);
  }
  else
  {
    STKUNIT_ASSERT_LE(imbalance_threshold, 1.5);
  }

  // And verify that all dependent entities are on the same proc as their parent element
  {
    stk_classic::mesh::EntityVector entities;
    stk_classic::mesh::Selector selector1 = fem_meta.universal_part();

    get_selected_entities(selector1, bulk_data.buckets(element_rank), entities);
    bool result = stk_classic::rebalance::verify_dependent_ownership(NODE_RANK, entities);
    STKUNIT_ASSERT( result );
  }
}

/// \until rebalance::rebalance(

The test passes if the resulting rebalance produces an imbalance measure below 1.1 for 2 or 4 procs and below 1.5 otherwise.

See UnitTestZoltanGraph.cpp for the complete source listing.

Simple Zoltan Unit Test

This unit test creates a 2D quad mesh on proc 0 with coordinates and Parts associated with edges, nodes, and constraints and then moves these entities as determined by calling Zoltan's load balancing capability using default settings.

Using the following mesh of 4 quads,

Global node and element numbering

      7       8       9
      +-------+-------+
      |       |       |
      |  e3   |  e4   |
      |       |5      |
     4+-------+-------+6   
      |       |       |     Y
      |  e1   |  e2   |     |
      |       |       |     |
      +-------+-------+     *--> X
      1       2      3

Local node numbering

      3       4
      +-------+
      |       |
      |  e1   |
      |       |
      +-------+
      1       2

Use of Zoltan with default settings is achieved by instantiating the appropriate Partition class with an empty parameter list as follows,

  // Zoltan partition is specialized fomm a virtual base class, stk_classic::rebalance::Partition.
  // Other specializations are possible.
  Teuchos::ParameterList emptyList;
  stk_classic::rebalance::Zoltan zoltan_partition(comm, spatial_dimension, emptyList);

An initial assessment of imbalance is made using an element weight field followed by a call to actually do the rebalance as follows,

/// \until rebalance::rebalance(

Perfect balancing should result using 2 or 4 procs, and on 3 procs, the imbalance threshold should be below 1.5. The test passes if these criteria are satisfied.

See UnitTestZoltanSimple.cpp for the complete source listing.