Analysis of Inverted Generational Distance¶
This page demonstrates how optimization algorithm benchmarking data can be loaded from disk, metrics (such as inverted generational distance) can be calculated for all reported populations, and the resulting data analyzed for comparisons between experiements. We demonstrate the process by comparing three algorithms against eachother on a single test problem. The resulting comparisons are displayed and it is shown how the table can be exported to latex for publication.
In [1]:
Copied!
import paretobench as pb
import paretobench as pb
In [2]:
Copied!
%%time
# Evaluate the IGD metric on all populations stored in three saved experiments
df = pb.eval_metrics_experiments(
["../data/NSGAII.h5", "../data/CPSMOEA.h5", "../data/ParEGO.h5"],
pb.InvertedGenerationalDistance(),
n_procs=8,
)
df.head()
%%time
# Evaluate the IGD metric on all populations stored in three saved experiments
df = pb.eval_metrics_experiments(
["../data/NSGAII.h5", "../data/CPSMOEA.h5", "../data/ParEGO.h5"],
pb.InvertedGenerationalDistance(),
n_procs=8,
)
df.head()
<timed exec>:2: DeprecationWarning: eval_metrics_experiments is deprecated and will be removed in a future release of ParetoBench. Please use eval_metrics instead
CPU times: user 232 ms, sys: 89.8 ms, total: 322 ms Wall time: 2.69 s
Out[2]:
| problem | fevals | run_idx | pop_idx | exp_name | exp_idx | fname | igd | |
|---|---|---|---|---|---|---|---|---|
| 0 | WFG1 (m=2,n=16,k=2) | 20 | 0 | 0 | NSGA-II | 0 | ../data/NSGAII.h5 | 2.207231 |
| 1 | WFG1 (m=2,n=16,k=2) | 40 | 0 | 1 | NSGA-II | 0 | ../data/NSGAII.h5 | 1.545975 |
| 2 | WFG1 (m=2,n=16,k=2) | 60 | 0 | 2 | NSGA-II | 0 | ../data/NSGAII.h5 | 1.530251 |
| 3 | WFG1 (m=2,n=16,k=2) | 80 | 0 | 3 | NSGA-II | 0 | ../data/NSGAII.h5 | 1.529974 |
| 4 | WFG1 (m=2,n=16,k=2) | 100 | 0 | 4 | NSGA-II | 0 | ../data/NSGAII.h5 | 1.528507 |
In [3]:
Copied!
# Calculate aggregate statistics of the metric on each history object including individuals up until a budget of `max_feval`
# function evaluations has been exceeded. Use the Wilcoxon rank sum test to compare algorithms against ParEGO.
dfa = pb.aggregate_metrics_feval_budget(df, max_feval=300, wilcoxon_idx=2)
dfa.head()
# Calculate aggregate statistics of the metric on each history object including individuals up until a budget of `max_feval`
# function evaluations has been exceeded. Use the Wilcoxon rank sum test to compare algorithms against ParEGO.
dfa = pb.aggregate_metrics_feval_budget(df, max_feval=300, wilcoxon_idx=2)
dfa.head()
/opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/paretobench/stats.py:49: RuntimeWarning: invalid value encountered in scalar divide z = (s - expected) / np.sqrt(n1 * n2 * (n1 + n2 + 1) / 12.0) /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/paretobench/stats.py:49: RuntimeWarning: invalid value encountered in scalar divide z = (s - expected) / np.sqrt(n1 * n2 * (n1 + n2 + 1) / 12.0)
Out[3]:
| igd | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| mean | std | median | pct_2.5 | pct_97.5 | wilcoxon_best | wilcoxon_comp | ||||
| problem | exp_idx | fname | exp_name | |||||||
| WFG1 (k=2, m=2, n=16) | 0 | ../data/NSGAII.h5 | NSGA-II | 1.472478 | 0.284833 | 1.370514 | 1.200316 | 2.059161 | False | - |
| 1 | ../data/CPSMOEA.h5 | CPS-MOEA | 1.359804 | 0.014544 | 1.362041 | 1.333089 | 1.376491 | False | - | |
| 2 | ../data/ParEGO.h5 | ParEGO | 1.318559 | 0.004923 | 1.319673 | 1.307050 | 1.324377 | True | ||
| WFG1 (k=2, m=2, n=32) | 0 | ../data/NSGAII.h5 | NSGA-II | 1.535811 | 0.304517 | 1.430068 | 1.267754 | 2.148538 | False | - |
| 1 | ../data/CPSMOEA.h5 | CPS-MOEA | 1.369364 | 0.009933 | 1.368849 | 1.357025 | 1.386730 | False | - | |
In [4]:
Copied!
# Turn the aggregate stats into a table comparing metric values for each problem between algorithms.
dfc = pb.construct_metric_comparison_table(
dfa,
problem_params="n",
mean_fmt_kwargs={"precision": 3, "exp_digits": 1},
std_fmt_kwargs={"precision": 3, "exp_digits": 1},
)
dfc
# Turn the aggregate stats into a table comparing metric values for each problem between algorithms.
dfc = pb.construct_metric_comparison_table(
dfa,
problem_params="n",
mean_fmt_kwargs={"precision": 3, "exp_digits": 1},
std_fmt_kwargs={"precision": 3, "exp_digits": 1},
)
dfc
Out[4]:
| CPS-MOEA | NSGA-II | ParEGO | ||
|---|---|---|---|---|
| Problem | n | |||
| WFG1 | 16 | 1.360e+0 (1.454e-2) - | 1.472e+0 (2.848e-1) - | \cellbold 1.319e+0 (4.923e-3) |
| 32 | 1.369e+0 (9.933e-3) - | 1.536e+0 (3.045e-1) - | \cellbold 1.322e+0 (3.700e-3) | |
| 64 | \cellbold 1.374e+0 (7.857e-3) = | 1.494e+0 (2.471e-1) = | - |
In [5]:
Copied!
# Demonstrate printing to latex
print(pb.comparison_table_to_latex(dfc))
# Demonstrate printing to latex
print(pb.comparison_table_to_latex(dfc))
\begin{tabular}{lllll}
\toprule
\multicolumn{1}{c}{\textbf{Problem }} &\multicolumn{1}{c}{\textbf{ n }} & \multicolumn{1}{c}{\textbf{ CPS-MOEA }}& \multicolumn{1}{c}{\textbf{ NSGA-II }}& \multicolumn{1}{c}{\textbf{ ParEGO }}\\
\midrule
\multirow{3}{*}{WFG1} & 16 & 1.360e+0 (1.454e-2) - & 1.472e+0 (2.848e-1) - & \cellbold 1.319e+0 (4.923e-3) \\
& 32 & 1.369e+0 (9.933e-3) - & 1.536e+0 (3.045e-1) - & \cellbold 1.322e+0 (3.700e-3) \\
& 64 & \cellbold 1.374e+0 (7.857e-3) $\approx$ & 1.494e+0 (2.471e-1) $\approx$ & - \\
\cline{1-5}
\multicolumn{2}{c}{+/-/$\approx$} & \multicolumn{1}{c}{0/2/1} & \multicolumn{1}{c}{0/2/1} & \\
\bottomrule
\end{tabular}
In [ ]:
Copied!