Manak Hierarchical Structure

MANAK hierarchical structure -

  Master Suite
        |
        |
   Manak Suite
        |
        |
  Benchmark Case
        |
        |
Sub-Benchmark Case

Master Suite

When you include manak.hpp, it also adds a master suite to the program. Master suite is ancestor of all benchmark suites and cases. The 'main' function of Manak always runs this master suite which eventually runs the cases. Check out Set main function type for more information on main functions.

Manak Suite

Manak Suite is a collection of Manak suites and cases which can be accessed by a unique name. For example,

  Master Suite
        |
        |
     Suite1
     |    |
     |    |
   Suite2 |
          |
        Case1

Here, Suite1 is a direct child of master suite having name /Suite1. Suite2 will have name /Suite1/Suite2. Specific benchmarking suites can be run with regular expression support provided by Manak. Check out Running Specific Cases for regular expression support.

Benchmark Case

Like benchmark suites, benchmarking cases in Manak are also accessed by unique hierarchical names. Benchmarking cases in Manak can also perform testing with 'TEST' blocks. Check out Test Framework for more information on 'TEST' blocks. Manak also supports running specific case with regular expressions(see Running Specific Cases). For comparison, cases can have muliple entries under different library names. For example -

#define MANAK_MODULE Bench
#define MANAK_AUTO_MAIN
#define MANAK_INIT

#include <manak/manak.hpp>

MANAK_AUTO_BENCHMARK_CASE(Test, Lib1)
{
  MEASURE
  (
    for(size_t i = 0;i < 1000;i++);
  )
}

MANAK_AUTO_BENCHMARK_CASE(Test, Lib2)
{
  MEASURE
  (
    for(size_t i = 0;i < 10000;i++);
  )
}

NOTE: 'MANAK_AUTO_MAIN' or 'MANAK_INIT' has to be defined only once in case of multi file project. Check out Using Manak Effectively in Multi File Project for more information.

In the above example there are 2 entries under Benchmark Case 'Test' with libraries 'Lib1' and 'Lib2'. Check Sample HTML ouput for output of above code. In the output you will notice that library entries are compared against each other. Check out Comparison Framework for more information on comparison framework.

Sub-Benchmark Case

Sub-Benchmark cases are only used in case of parametrized benchmarks. Check out Complete guide on Parametrized Benchmarking for guide on parametrized benchmarking.

Sub-Benchmark case may be the smallest unit in this hierarchy but it is the most important and complex. There can be any number of sub-benchmark cases in a benchmark case and they are identified with index rather than a name(although a name can be assigned). Index is provided based on the chronological order(starting with 0). So when multiple library entries of benchmark case are compared against each other(see Comparison Framework if you don't know when they are compared), the corresponding sub-benchmark cases are compared against each other with index. The benchmark cases in comparison can have different number of sub-benchmark cases. For example -

#include <iostream>

#define MANAK_MODULE Test

#define MANAK_INIT
#define MANAK_AUTO_MAIN

#include <manak/manak.hpp>

using namespace std;

void ForLoops(size_t a)
{
  MEASURE
  (
    for(size_t i = 0;i < a;i++);
  )
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, Lib1, ForLoops)->AddArgs_N("With100", 100)->AddArgs_N("With1000", 1000)->AddArgs_N("With10000", 10000));

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, Lib2, ForLoops)->AddArgs_N("With100", 100)->AddArgs_N("With 1000", 1000));

NOTE: 'MANAK_AUTO_MAIN' or 'MANAK_INIT' has to be defined only once in case of multi file project. Check out Using Manak Effectively in Multi File Project for more information.

The output of above code - Sample HTML ouput. Check out HTML for information on HTML output. You can see here that case 'B1' with Lib1 contains 3 sub-cases while with Lib2 contains 2 sub-cases. Take a look at the detailed report, in B1-lib1 all the sub-cases are named but the second one does not reflect into the main table of 'B1'. This is because, second sub-case name of lib1 and second sub-case name of lib2 do not match(there is an extra space). As sub-cases are characterized only by their index, names of sub-cases only reflect in the main table if they match for every library.

#include <iostream>

#define MANAK_MODULE Test

#define MANAK_INIT
#define MANAK_AUTO_MAIN

#include <manak/manak.hpp>

using namespace std;

void ForLoops(size_t a)
{
  MEASURE
  (
    for(size_t i = 0;i < a;i++);
  )
}

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, Lib1, ForLoops)->AddArgs_N("With100", 100)->AddArgs_N("With1000", 1000)->AddArgs_N("With10000", 10000));

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B1, Lib2, ForLoops)->AddArgs_N("With100", 100)->AddArgs_N("With 1000", 1000));

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B2, Lib1, ForLoops)->AddArgs_N("With100", 100)->AddArgs_N("With1000", 1000)->AddArgs_N("With10000", 10000));

MANAK_ADD_CASE(MANAK_CREATE_BENCHMARK_WITH_TEMPLATE(B2, Lib2, ForLoops)->AddArgs_N("With100", 100)->AddArgs_N("With1000", 1000));

Check out the output of the above code HERE

This also applies to TXT output.