BENCH

Introduction

BENCH is a GAMS solver to help facilitate benchmarking of GAMS optimization solvers. BENCH calls all user-specified GAMS solvers for a particular modeltype and captures results in the standard GAMS listing file format. BENCH can call the GAMS/EXAMINER solver automatically to independently verify feasibility and optimality of the solution returned.

There are several advantages to using the BENCH solver instead of just creating a GAMS model or batch file to do multiple solves with the various solvers. The first is that the model does not need to be generated individually for each solve; BENCH spawns each solver using the matrix file generated during the initial call to BENCH. Furthermore, BENCH simplifies solution examination/verification by automatically utilizing EXAMINER. And finally, data can automatically be collected for use with the PAVER performance analysis server.

BENCH comes free with any GAMS system. Licensing is dependent on licensing of the subsolvers. Thus, BENCH runs all solvers for which the user has a valid license.

How to run a Model with BENCH:

BENCH is run like any other GAMS solver. From the command line this is:

>> gams modelname modeltype=bench

where modelname is the GAMS model name and modeltype the solver indicator for a particular model type (e.g. LP, MIP, RMIP, QCP, MIQCP, RMIQCP, NLP, DNLP, CNS, MINLP, or MCP). BENCH can also be specified via the option statement within the model itself before the solve statement:

option modeltype=bench;

The user must specify the solvers to be included by using the solvers option (specified in a solver option file called bench.opt). Otherwise, GAMS/BENCH returns with a warning message

Warning: no solvers selected. Nothing to be done.

For more information on using solver option files and the solvers option, see Section User-Specified Options.

User-Specified Options

GAMS Options

BENCH works like other GAMS solvers, and many options can be set in the GAMS model. The most relevant GAMS options are nodlim, optca, optcr, optfile, cheat, cutoff, prioropt, and tryint. These options are global in the sense that they are passed on to all subsolvers called.

The options can be set either through an option statement

option optfile=1;

or through a model suffix, which sets them only for an individual model

modelname.optfile=1;

All solver-related options are implemented in BENCH and are passed on to the respective solvers. We remark that for a particular subsolver some of these options may not be valid. In this case, although they are passed on by BENCH to the respective subsolver, they may not be used.

The options listed below differ from the usual implementation and are based on individual limits for each solver called by BENCH.

Option Description Default
iterlim Sets the individual iteration limit. The subsolver called by BENCH will terminate and pass on the current solution if the number of iterations for each solver exceeds this limit. 2000000000
reslim Sets the individual time limit in seconds. The subsolver called by BENCH will terminate and pass on the current solution if the resource time for each solver exceeds this limit. 1000

The BENCH Options

BENCH solver options are passed on through solver option files. If you specify <modelname>.optfile = 1; before the SOLVE statement in your GAMS model. BENCH will then look for and read an option file with the name bench.opt (see The Solver Option File for general use of solver option files). Unless explicitly specified in the BENCH option file, the solvers called by BENCH will not read option files. The syntax for the BENCH option file is

optname value

with one option on each line.

For example,

solvers  conopt.1 minos snopt.2

This option determines the solvers to be called and is required. If the solvers option is omitted, then BENCH terminates with a warning message.

In this example, CONOPT will be called first with the option file conopt.opt. Then MINOS will be called with no option file and SNOPT will be called last with the option file snopt.op2. We note that the solvers are called in this order. This can be of particular use since detailed solution information at the end of the GAMS listing file is for the final solver called. The input of the solver option file is echoed in the listing file created by BENCH to help distinguish the different solver calls. See the section describing the BENCH listing file for details.

Specifying separate solver option files can also be useful to specify the subsolver that is to be used by a solver.

Option Description Default
allsolvers Indicator whether all valid solvers for given modeltype should be run. 0
dualcstol Tolerance on complementary slackness between dual variables and the primal constraints: passed to EXAMINER 1e-7
dualfeastol Dual variables and constraints feasibility tolerance: passed to EXAMINER 1e-6
examiner Indicator whether to call GAMS/EXAMINER to independently verify solution for feasibility and optimality. 0
outlev Log output level.
1 BENCH summary log output only
2 BENCH summary log output and log output of each solver
2
paver Indicator whether PAVER trace files should be written.
Enabling causes a trace file solver.pvr to be written for each solver called. If the solver uses an option file, then the resulting file is solver-optnum.pvr, where optnum is the option file number. The files created can be submitted to the PAVER Server for automated performance analysis.
0
paverex Indicator whether PAVER trace files should be written for the Examiner run.
Enabling causes a trace file solver-ex.pvr to be written for each solver called. If any Examiner check fails (independent or solver), then we return a model status 14 (no solution returned) and a solver status of 4 (terminated by solver). If no Examiner check is done, for example, because the return status is infeasible, then the status codes are returned as is. If the solver uses an option file, then the resulting file is solver-optnum-ex.pvr, where optnum is the option file number. The files created can be submitted to the PAVER Server for automated performance analysis.
0
primalcstol Tolerance on complementary slackness between primal variables and the dual constraints: passed to EXAMINER 1e-7
primalfeastol Primal variables and constraints feasibility tolerance: passed to EXAMINER 1e-6
returnlastsol Indicator whether to return solution from the last solver 0
solvers List of solvers to benchmark.
solver[.n] gives the name of the GAMS solver that should be used, where n is the integer corresponding to the options file. If .n is missing, the solver will not look for an options file. This is a required option.

Benchmark Analysis Using the PAVER Server

Benchmark data obtained using GAMS/BENCH can be automatically analyzed using the PAVER Server.

In order to enable creation of the necessary data files for submission to PAVER, users must enable the paver option.

For example, suppose a user has a set of models and wishes to compare three solvers, say CONOPT3, MINOS, and SNOPT. The user would then create a bench.opt solver option file with the entries

solvers conopt3 minos snopt
paver 1

Solving the models using bench as the solver will create PAVER data files, namely one for each solver: conopt.pvr, minos.pvr, and snopt.pvr, which can be submitted to the PAVER server at

http://www.gamsworld.org/performance/paver/pprocess_submit.htm

for automated analysis. Note that all PAVER trace files are appended to if they exist and if subsequent solves are made.

Solution Verification Using Examiner

Examiner Checks

BENCH can automatically call the GAMS/EXAMINER solver to check the solution for feasibility and complementarity. In particular, EXAMINER checks for

  • primal feasibility: feasibility of both primal variables and primal constraints.
  • dual feasibility: feasibility of both dual variables and dual constraints.
  • primal complementary slackness: complementary slackness of primal variables to dual constraints.
  • dual complementary slackness: complementary slackness of dual variables to primal constraints.

where EXAMINER does two types of checks:

  • Solvepoint: the point returned by the solver. The solver returns both level and marginal values for the rows and columns: Examiner uses these exactly as given.
  • Solupoint: EXAMINER uses the variable levels (primal variables) and equation marginals (dual variabless) to compute the equation levels and variable marginals. The variable levels and equation marginals used are those returned by the subsolver.

By default, BENCH does not call EXAMINER to verify the solution. To enable solution verification, specify

examiner 1

in the bench.opt solver option file. Of interest are also the EXAMINER tolerances dualcstol, dualfeastol, primalcstol, and primalfeastol, which can also be set in the BENCH solver option file. For more information, see the EXAMINER documentation.

Examiner Output in BENCH

Examiner output, if solution verification is enabled, is given in the log output during the actual solve and summary information is given in the final BENCH summary under the Examiner column. Models either pass (P) or fail (F) based on the default Examiner or user-specified tolerances given. If EXAMINER does not do a check, for example, because the solver returns a model status of infeasible, then the Examiner column is given as (N).

If Examiner is not enabled, then n/a is listed under the Examiner column.

The first entry under Examiner is the Examiner status for using solver provided variable constraint level values (solvpoint). The second entry is the solupoint, where GAMS computes the constraint levels from the variable levels returned by the solver.

An example is given below, where we specified to use the solvers BDMLP, MINOS, XPRESS, and CPLEX on the GAMS Model Library model trnsport:

Solver     Modstat  Solstat    Objective       ResUsd    Examiner
-----------------------------------------------------------------
BDMLP          1       1        153.6750        0.000       P/P
MINOS          1       1        153.6750        0.000       P/P
XPRESS         1       1        153.6750        0.040       P/P
CPLEX          1       1        153.6750        0.000       P/P

In the example below, EXAMINER is enabled, but does not perform any checks because the return status of the solver lists the model as infeasible (see the Examiner column (N/N)).

Solver     Modstat  Solstat    Objective       ResUsd    Examiner
-----------------------------------------------------------------
BDMLP          5       1          0.0000        0.000       N/N

For models having discrete variables, for example MIP, MIQCP, or MINLP, we also show the best bound. A sample output using the GAMS model library model magic is shown below.

Solver    Modstat Solstat      Objective      BestBound      ResUsd  Examiner
-----------------------------------------------------------------------------
CPLEX           8       1    991970.0000    985514.2857       0.000      n/a
XPRESS          1       1    988540.0000    988540.0000       0.060      n/a
MOSEK           8       1    988540.0000    988540.0000       0.170      n/a

Output

The BENCH Log File

The BENCH log output contains complete log information for each solver called. The individual solver calls are indicated by the entry

--- Spawning solver : (Solver Name)

followed by the log output of the individual solver.

An example of the log output using the transportation model trnsport) from the GAMS model library. We specify the solvers BDMLP, XPRESS, MINOS and CPLEX via the option file bench.opt:

GAMS Rev 138  Copyright (C) 1987-2004 GAMS Development. All rights reserved
Licensee: GAMS Development Corp.                         G040421:1523CR-LNX
          GAMS Development Corp.                                     DC3665
--- Starting compilation
--- trnsport.gms(69) 3 Mb
--- Starting execution
--- trnsport.gms(45) 4 Mb
--- Generating model transport
--- trnsport.gms(66) 4 Mb
---    6 rows, 7 columns, and 19 non-zeroes.
--- Executing BENCH

GAMS/BENCH    Jan 19, 2004 LNX.00.NA 21.3 004.027.041.LXI

  GAMS Benchmark Solver

  Reading user supplied options file /home/gams/support/bench.opt
  Processing...
  > solvers bdmlp minos xpress cplex


--- Spawning solver : BDMLP


 BDMLP 1.3     Jan 19, 2004 LNX.00.01 21.3 058.050.041.LXI

 Reading data...
 Work space allocated           --    0.03 Mb

    Iter    Sinf/Objective  Status    Num  Freq
       1    2.25000000E+02  infeas      1     1
       4    1.53675000E+02  nopt        0
 SOLVER STATUS: 1 NORMAL COMPLETION
 MODEL STATUS : 1 OPTIMAL
 OBJECTIVE VALUE     153.67500

--- Spawning solver : MINOS


MINOS-Link    Jan 19, 2004 LNX.M5.M5 21.3 029.050.041.LXI GAMS/MINOS 5.51

    GAMS/MINOS 5.51, Large Scale Nonlinear Solver
    B. A. Murtagh, University of New South Wales
    P. E. Gill, University of California at San Diego,
    W. Murray, M. A. Saunders, and M. H. Wright,
    Systems Optimization Laboratory, Stanford University

 Work space allocated           --    1.01 Mb

  Reading Rows...
  Reading Columns...

 EXIT - Optimal Solution found, objective:        153.6750


--- Spawning solver : XPRESS


Xpress-MP     Jan 19, 2004 LNX.XP.XP 21.3 024.027.041.LXI Xpress lib 14.24
Xpress-MP licensed by Dash to GAMS Development Corp. for GAMS

Reading data . . . done.

Reading Problem gmsxp_xx
Problem Statistics
           6 (      0 spare) rows
           7 (      0 spare) structural columns
          19 (      0 spare) non-zero elements
Global Statistics
           0 entities        0 sets        0 set members
Presolved problem has:      5 rows       6 cols      12 non-zeros

   Its         Obj Value      S   Ninf  Nneg        Sum Inf  Time
     0           .000000      D      3     0     900.000000     0
     3        153.675000      D      0     0        .000000     0
Uncrunching matrix
     3        153.675000      D      0     0        .000000     0
Optimal solution found
optimal LP solution found: objective value 153.675

--- Spawning solver : CPLEX


GAMS/Cplex    Jan 19, 2004 LNX.CP.CP 21.3 025.027.041.LXI For Cplex 9.0
Cplex 9.0.0, GAMS Link 25

Reading data...
Starting Cplex...
Tried aggregator 1 time.
LP Presolve eliminated 1 rows and 1 columns.
Reduced LP has 5 rows, 6 columns, and 12 nonzeros.
Presolve time =    0.00 sec.

Iteration      Dual Objective            In Variable           Out Variable
     1              73.125000    x(seattle.new-york) demand(new-york) slack
     2             119.025000     x(seattle.chicago)  demand(chicago) slack
     3             153.675000    x(san-diego.topeka)   demand(topeka) slack
     4             153.675000  x(san-diego.new-york)  supply(seattle) slack

Optimal solution found.
Objective :         153.675000



--- BENCH SUMMARY:

  Solver     Modstat  Solstat    Objective       ResUsd    Examiner
  -----------------------------------------------------------------
  BDMLP          1       1        153.6750        0.000       n/a
  MINOS          1       1        153.6750        0.000       n/a
  XPRESS         1       1        153.6750        0.040       n/a
  CPLEX          1       1        153.6750        0.000       n/a

--- Restarting execution
--- trnsport.gms(66) 0 Mb
--- Reading solution for model transport
--- trnsport.gms(68) 3 Mb
*** Status: Normal completion

The BENCH Listing File

The BENCH listing file is similar to the standard GAMS format. It contains a benchmark summary of all solvers called. The regular solve summary for BENCH does not return a solution (although the solution of the final solve can be returned using the returnlastsol option). For the example below we use the batchdes model from the GAMS model library using the solvers SBB and DICOPT. We specify the two solvers using a bench.opt option file with the entry:

solvers sbb.1 dicopt

Note that SBB will use a solver option file called sbb.opt.

               S O L V E      S U M M A R Y

     MODEL   batch               OBJECTIVE  cost
     TYPE    MINLP               DIRECTION  MINIMIZE
     SOLVER  BENCH               FROM LINE  183

   **** SOLVER STATUS     1 NORMAL COMPLETION
   **** MODEL STATUS      14 NO SOLUTION RETURNED
   **** OBJECTIVE VALUE                0.0000

   RESOURCE USAGE, LIMIT          0.000      1000.000
   ITERATION COUNT, LIMIT         0         10000
     Reading user supplied options file /home/models/bench.opt
     Processing...
     > solvers sbb.1 dicopt

Note that the model status return code for BENCH itself is always SOLVER STATUS 1 and MODEL STATUS 14, since BENCH itself does not return a solution by default. To obtain the status codes and solution information of the last solver, the returnlastsol option can be enabled.

In addition the listing file contains complete solve summary information for each solver called. Also, note that the option file used for SBB and its contents are echoed to the SBB summary.

                 B E N C H M A R K      S U M M A R Y


       SOLVER             SBB
       SOLVER STATUS    1 NORMAL COMPLETION
       MODEL STATUS     8 INTEGER SOLUTION
       OBJECTIVE VALUE           167427.6571
       RESOURCE USAGE, LIMIT          0.080       1000.000
       ITERATION COUNT, LIMIT       139         100000
       EVALUATION ERRORS, LIMIT       0              0
       OPTION FILE        sbb.opt

    Reading user supplied options file sbb.opt
    Processing...
    > rootsolver conopt2
    > subsolver snopt



       SOLVER             DICOPT
       SOLVER STATUS    1 NORMAL COMPLETION
       MODEL STATUS     8 INTEGER SOLUTION
       OBJECTIVE VALUE           167427.6571
       RESOURCE USAGE, LIMIT          0.100        999.920
       ITERATION COUNT, LIMIT       117          99861
       EVALUATION ERRORS, LIMIT       0              0

Note that the listing file does not contain detailed solution information since BENCH does not return any values.

Interrupting BENCH with Ctrl-C

BENCH passes all Control-C (Ctrl-C) signals to the respective subsolvers. If a terminate signal via Ctrl-C is sent in the middle of a solver run (i.e. not initially when the solver begins execution), the individual subsolver is terminated.

To terminate not only the subsolver but also BENCH, a Ctrl-C signal should be sent at the beginning of a solver's execution. Thus, several Ctrl-C in rapid succession will terminate BENCH.

Benchmark summary information will be written to the listing file for each solver that has successfully completed without any signal interrupt.

Benchmark Example

In this section we will give a small example showing how to use the BENCH solver and automate the subsequent analysis using the PAVER Server. In particular, we will run the three versions of CONOPT (CONOPT1, CONOPT2, and CONOPT3, as well as CONOPT3 with no scaling) on the default instance of the COPS models for nonlinear programming. We will use the 17 models available from the GAMS Model Library.

Firs we need to extract all of the models from the GAMS Model Library. We can create a file which will extract these automatically. Create a file called getcops.gms with the entries below:

$call gamslib camshape
$call gamslib catmix
$call gamslib chain
$call gamslib elec
$call gamslib flowchan
$call gamslib gasoil
$call gamslib glider
$call gamslib jbearing
$call gamslib lnts
$call gamslib methanol
$call gamslib minsurf
$call gamslib pinene
$call gamslib polygon
$call gamslib popdynm
$call gamslib robot
$call gamslib rocket
$call gamslib torsion

Running the file using gams getcops.gms extracts the models. Then create a BENCH solver option file called bench.opt with the entries

solvers conopt1 conopt2 conopt3 conopt3.1
paver 1

The first entry tells BENCH to run the solvers CONOPT1, CONOPT2, and CONOPT3 and then CONOPT3 with the option file conopt3.opt. The second entry tells BENCH to create PAVER trace files. These can be submitted to the PAVER server for automated performance analysis. Now create an option file conopt3.opt with the entry

lsscal f

which tells CONOPT3 not to use scaling.

We can now run the models in batch mode, for example by creating a GAMS batch file runcops.gms with the following entries:

$call gams camshape.gms nlp=bench optfile=1 reslim=10 domlim=99999
$call gams catmix.gms   nlp=bench optfile=1 reslim=10 domlim=99999
$call gams chain.gms    nlp=bench optfile=1 reslim=10 domlim=99999
$call gams elec.gms     nlp=bench optfile=1 reslim=10 domlim=99999
$call gams flowchan.gms nlp=bench optfile=1 reslim=10 domlim=99999
$call gams gasoil.gms   nlp=bench optfile=1 reslim=10 domlim=99999
$call gams glider.gms   nlp=bench optfile=1 reslim=10 domlim=99999
$call gams jbearing.gms nlp=bench optfile=1 reslim=10 domlim=99999
$call gams lnts.gms     nlp=bench optfile=1 reslim=10 domlim=99999
$call gams methanol.gms nlp=bench optfile=1 reslim=10 domlim=99999
$call gams minsurf.gms  nlp=bench optfile=1 reslim=10 domlim=99999
$call gams pinene.gms   nlp=bench optfile=1 reslim=10 domlim=99999
$call gams polygon.gms  nlp=bench optfile=1 reslim=10 domlim=99999
$call gams popdynm.gms  nlp=bench optfile=1 reslim=10 domlim=99999
$call gams robot.gms    nlp=bench optfile=1 reslim=10 domlim=99999
$call gams rocket.gms   nlp=bench optfile=1 reslim=10 domlim=99999
$call gams torsion.gms  nlp=bench optfile=1 reslim=10 domlim=99999

Running the file using the command gams runcops.gms runs all models with all three solvers through the GAMS/BENCH solver. Furthermore, three PAVER trace files are created: conopt1.pvr, conopt2.pvr, conopt3.pvr and conopt3-1.pvr, where the latter is for CONOPT3 with no scaling. Users can then submit the three trace files to the PAVER Server for automated analysis.

The resulting performance plot in the following figure shows the efficiency of each solver/solver option.

conopt.png
PAVER: Process Overview