Table of Contents
- User-Specified Options
- Benchmark Analysis Using the PAVER Server
- Solution Verification Using Examiner
- Interrupting BENCH with Ctrl-C
- Benchmark Example
BENCH is a GAMS solver to help facilitate benchmarking of GAMS optimization solvers. BENCH calls all user-specified GAMS solvers for a particular modeltype and captures results in the standard GAMS listing file format. BENCH can call the GAMS/EXAMINER solver automatically to independently verify feasibility and optimality of the solution returned.
There are several advantages to using the BENCH solver instead of just creating a GAMS model or batch file to do multiple solves with the various solvers. The first is that the model does not need to be generated individually for each solve; BENCH spawns each solver using the matrix file generated during the initial call to BENCH. Furthermore, BENCH simplifies solution examination/verification by automatically utilizing EXAMINER. And finally, data can automatically be collected for use with the PAVER performance analysis server.
BENCH comes free with any GAMS system. Licensing is dependent on licensing of the subsolvers. Thus, BENCH runs all solvers for which the user has a valid license.
BENCH is run like any other GAMS solver. From the command line this is:
>> gams modelname modeltype=bench
modelname is the GAMS model name and
modeltype the solver indicator for a particular model type (e.g. LP, MIP, RMIP, QCP, MIQCP, RMIQCP, NLP, DNLP, CNS, MINLP, or MCP). BENCH can also be specified via the option statement within the model itself before the solve statement:
The user must specify the solvers to be included by using the
solvers option (specified in a solver option file called
bench.opt). Otherwise, GAMS/BENCH returns with a warning message
Warning: no solvers selected. Nothing to be done.
For more information on using solver option files and the
solvers option, see Section User-Specified Options.
BENCH works like other GAMS solvers, and many options can be set in the GAMS model. The most relevant GAMS options are
nodlim, optca, optcr, optfile, cheat, cutoff, prioropt, and
tryint. These options are global in the sense that they are passed on to all subsolvers called.
The options can be set either through an option statement
or through a model suffix, which sets them only for an individual model
All of the options listed in the Chapter Basic Solver Usage are implemented in BENCH and are passed on to the respective solvers. We remark that for a particular subsolver some of these options may not be valid. In this case, although they are passed on by BENCH to the respective subsolver, they may not be used.
The options listed below differ from the usual implementation and are based on individual limits for each solver called by BENCH.
|Sets the individual iteration limit. The subsolver called by BENCH will terminate and pass on the current solution if the number of iterations for each solver exceeds this limit.|
|Sets the individual time limit in seconds. The subsolver called by BENCH will terminate and pass on the current solution if the resource time for each solver exceeds this limit.|
BENCH solver options are passed on through solver option files. If you specify
<modelname>.optfile = 1; before the SOLVE statement in your GAMS model. BENCH will then look for and read an option file with the name
bench.opt (see The Solver Option File for general use of solver option files). Unless explicitly specified in the BENCH option file, the solvers called by BENCH will not read option files. The syntax for the BENCH option file is
with one option on each line.
solvers conopt.1 minos snopt.2
This option determines the solvers to be called and is required. If the solvers option is omitted, then BENCH terminates with a warning message.
In this example, CONOPT will be called first with the option file
conopt.opt. Then MINOS will be called with no option file and SNOPT will be called last with the option file
snopt.op2. We note that the solvers are called in this order. This can be of particular use since detailed solution information at the end of the GAMS listing file is for the final solver called. The input of the solver option file is echoed in the listing file created by BENCH to help distinguish the different solver calls. See the section describing the BENCH listing file for details.
Specifying separate solver option files can also be useful to specify the subsolver that is to be used by a solver.
|allsolvers||Indicator whether all valid solvers for given modeltype should be run.|
|dualcstol||Tolerance on complementary slackness between dual variables and the primal constraints: passed to EXAMINER|
|dualfeastol||Dual variables and constraints feasibility tolerance: passed to EXAMINER|
|examiner||Indicator whether to call GAMS/EXAMINER to independently verify solution for feasibility and optimality.|
|outlev||Log output level.|
|paver||Indicator whether PAVER trace files should be written.|
Enabling causes a trace file
|paverex||Indicator whether PAVER trace files should be written for the Examiner run.|
Enabling causes a trace file
|primalcstol||Tolerance on complementary slackness between primal variables and the dual constraints: passed to EXAMINER|
|primalfeastol||Primal variables and constraints feasibility tolerance: passed to EXAMINER|
|returnlastsol||Indicator whether to return solution from the last solver|
|solvers||List of solvers to benchmark.|
Benchmark data obtained using GAMS/BENCH can be automatically analyzed using the PAVER Server.
In order to enable creation of the necessary data files for submission to PAVER, users must enable the paver option.
For example, suppose a user has a set of models and wishes to compare three solvers, say CONOPT3, MINOS, and SNOPT. The user would then create a
bench.opt solver option file with the entries
solvers conopt3 minos snopt paver 1
Solving the models using
bench as the solver will create PAVER data files, namely one for each solver:
snopt.pvr, which can be submitted to the PAVER server at
for automated analysis. Note that all PAVER trace files are appended to if they exist and if subsequent solves are made.
BENCH can automatically call the GAMS/EXAMINER solver to check the solution for feasibility and complementarity. In particular, EXAMINER checks for
- primal feasibility: feasibility of both primal variables and primal constraints.
- dual feasibility: feasibility of both dual variables and dual constraints.
- primal complementary slackness: complementary slackness of primal variables to dual constraints.
- dual complementary slackness: complementary slackness of dual variables to primal constraints.
where EXAMINER does two types of checks:
- Solvepoint: the point returned by the solver. The solver returns both level and marginal values for the rows and columns: Examiner uses these exactly as given.
- Solupoint: EXAMINER uses the variable levels (primal variables) and equation marginals (dual variabless) to compute the equation levels and variable marginals. The variable levels and equation marginals used are those returned by the subsolver.
By default, BENCH does not call EXAMINER to verify the solution. To enable solution verification, specify
bench.opt solver option file. Of interest are also the EXAMINER tolerances dualcstol, dualfeastol, primalcstol, and primalfeastol, which can also be set in the BENCH solver option file. For more information, see the EXAMINER documentation.
Examiner output, if solution verification is enabled, is given in the log output during the actual solve and summary information is given in the final BENCH summary under the
Examiner column. Models either pass (
P) or fail (
F) based on the default Examiner or user-specified tolerances given. If EXAMINER does not do a check, for example, because the solver returns a model status of infeasible, then the
Examiner column is given as (
If Examiner is not enabled, then
n/a is listed under the
The first entry under
Examiner is the Examiner status for using solver provided variable constraint level values (
solvpoint). The second entry is the
solupoint, where GAMS computes the constraint levels from the variable levels returned by the solver.
An example is given below, where we specified to use the solvers BDMLP, MINOS, XPRESS, and CPLEX on the GAMS Model Library model
Solver Modstat Solstat Objective ResUsd Examiner ----------------------------------------------------------------- BDMLP 1 1 153.6750 0.000 P/P MINOS 1 1 153.6750 0.000 P/P XPRESS 1 1 153.6750 0.040 P/P CPLEX 1 1 153.6750 0.000 P/P
In the example below, EXAMINER is enabled, but does not perform any checks because the return status of the solver lists the model as infeasible (see the
Examiner column (
Solver Modstat Solstat Objective ResUsd Examiner ----------------------------------------------------------------- BDMLP 5 1 0.0000 0.000 N/N
For models having discrete variables, for example MIP, MIQCP, or MINLP, we also show the best bound. A sample output using the GAMS model library model
magic is shown below.
Solver Modstat Solstat Objective BestBound ResUsd Examiner ----------------------------------------------------------------------------- CPLEX 8 1 991970.0000 985514.2857 0.000 n/a XPRESS 1 1 988540.0000 988540.0000 0.060 n/a MOSEK 8 1 988540.0000 988540.0000 0.170 n/a
The BENCH log output contains complete log information for each solver called. The individual solver calls are indicated by the entry
--- Spawning solver : (Solver Name)
followed by the log output of the individual solver.
An example of the log output using the transportation model
trnsport) from the GAMS model library. We specify the solvers
CPLEX via the option file
GAMS Rev 138 Copyright (C) 1987-2004 GAMS Development. All rights reserved Licensee: GAMS Development Corp. G040421:1523CR-LNX GAMS Development Corp. DC3665 --- Starting compilation --- trnsport.gms(69) 3 Mb --- Starting execution --- trnsport.gms(45) 4 Mb --- Generating model transport --- trnsport.gms(66) 4 Mb --- 6 rows, 7 columns, and 19 non-zeroes. --- Executing BENCH GAMS/BENCH Jan 19, 2004 LNX.00.NA 21.3 004.027.041.LXI GAMS Benchmark Solver Reading user supplied options file /home/gams/support/bench.opt Processing... > solvers bdmlp minos xpress cplex --- Spawning solver : BDMLP BDMLP 1.3 Jan 19, 2004 LNX.00.01 21.3 058.050.041.LXI Reading data... Work space allocated -- 0.03 Mb Iter Sinf/Objective Status Num Freq 1 2.25000000E+02 infeas 1 1 4 1.53675000E+02 nopt 0 SOLVER STATUS: 1 NORMAL COMPLETION MODEL STATUS : 1 OPTIMAL OBJECTIVE VALUE 153.67500 --- Spawning solver : MINOS MINOS-Link Jan 19, 2004 LNX.M5.M5 21.3 029.050.041.LXI GAMS/MINOS 5.51 GAMS/MINOS 5.51, Large Scale Nonlinear Solver B. A. Murtagh, University of New South Wales P. E. Gill, University of California at San Diego, W. Murray, M. A. Saunders, and M. H. Wright, Systems Optimization Laboratory, Stanford University Work space allocated -- 1.01 Mb Reading Rows... Reading Columns... EXIT - Optimal Solution found, objective: 153.6750 --- Spawning solver : XPRESS Xpress-MP Jan 19, 2004 LNX.XP.XP 21.3 024.027.041.LXI Xpress lib 14.24 Xpress-MP licensed by Dash to GAMS Development Corp. for GAMS Reading data . . . done. Reading Problem gmsxp_xx Problem Statistics 6 ( 0 spare) rows 7 ( 0 spare) structural columns 19 ( 0 spare) non-zero elements Global Statistics 0 entities 0 sets 0 set members Presolved problem has: 5 rows 6 cols 12 non-zeros Its Obj Value S Ninf Nneg Sum Inf Time 0 .000000 D 3 0 900.000000 0 3 153.675000 D 0 0 .000000 0 Uncrunching matrix 3 153.675000 D 0 0 .000000 0 Optimal solution found optimal LP solution found: objective value 153.675 --- Spawning solver : CPLEX GAMS/Cplex Jan 19, 2004 LNX.CP.CP 21.3 025.027.041.LXI For Cplex 9.0 Cplex 9.0.0, GAMS Link 25 Reading data... Starting Cplex... Tried aggregator 1 time. LP Presolve eliminated 1 rows and 1 columns. Reduced LP has 5 rows, 6 columns, and 12 nonzeros. Presolve time = 0.00 sec. Iteration Dual Objective In Variable Out Variable 1 73.125000 x(seattle.new-york) demand(new-york) slack 2 119.025000 x(seattle.chicago) demand(chicago) slack 3 153.675000 x(san-diego.topeka) demand(topeka) slack 4 153.675000 x(san-diego.new-york) supply(seattle) slack Optimal solution found. Objective : 153.675000 --- BENCH SUMMARY: Solver Modstat Solstat Objective ResUsd Examiner ----------------------------------------------------------------- BDMLP 1 1 153.6750 0.000 n/a MINOS 1 1 153.6750 0.000 n/a XPRESS 1 1 153.6750 0.040 n/a CPLEX 1 1 153.6750 0.000 n/a --- Restarting execution --- trnsport.gms(66) 0 Mb --- Reading solution for model transport --- trnsport.gms(68) 3 Mb *** Status: Normal completion
The BENCH listing file is similar to the standard GAMS format. It contains a benchmark summary of all solvers called. The regular solve summary for BENCH does not return a solution (although the solution of the final solve can be returned using the returnlastsol option). For the example below we use the
batchdes model from the GAMS model library using the solvers SBB and DICOPT. We specify the two solvers using a
bench.opt option file with the entry:
solvers sbb.1 dicopt
Note that SBB will use a solver option file called
S O L V E S U M M A R Y MODEL batch OBJECTIVE cost TYPE MINLP DIRECTION MINIMIZE SOLVER BENCH FROM LINE 183 **** SOLVER STATUS 1 NORMAL COMPLETION **** MODEL STATUS 14 NO SOLUTION RETURNED **** OBJECTIVE VALUE 0.0000 RESOURCE USAGE, LIMIT 0.000 1000.000 ITERATION COUNT, LIMIT 0 10000 Reading user supplied options file /home/models/bench.opt Processing... > solvers sbb.1 dicopt
Note that the model status return code for BENCH itself is always SOLVER STATUS 1 and MODEL STATUS 14, since BENCH itself does not return a solution by default. To obtain the status codes and solution information of the last solver, the returnlastsol option can be enabled.
In addition the listing file contains complete solve summary information for each solver called. Also, note that the option file used for SBB and its contents are echoed to the SBB summary.
B E N C H M A R K S U M M A R Y SOLVER SBB SOLVER STATUS 1 NORMAL COMPLETION MODEL STATUS 8 INTEGER SOLUTION OBJECTIVE VALUE 167427.6571 RESOURCE USAGE, LIMIT 0.080 1000.000 ITERATION COUNT, LIMIT 139 100000 EVALUATION ERRORS, LIMIT 0 0 OPTION FILE sbb.opt Reading user supplied options file sbb.opt Processing... > rootsolver conopt2 > subsolver snopt SOLVER DICOPT SOLVER STATUS 1 NORMAL COMPLETION MODEL STATUS 8 INTEGER SOLUTION OBJECTIVE VALUE 167427.6571 RESOURCE USAGE, LIMIT 0.100 999.920 ITERATION COUNT, LIMIT 117 99861 EVALUATION ERRORS, LIMIT 0 0
Note that the listing file does not contain detailed solution information since BENCH does not return any values.
BENCH passes all
Control-C (Ctrl-C) signals to the respective subsolvers. If a terminate signal via Ctrl-C is sent in the middle of a solver run (i.e. not initially when the solver begins execution), the individual subsolver is terminated.
To terminate not only the subsolver but also BENCH, a Ctrl-C signal should be sent at the beginning of a solver's execution. Thus, several Ctrl-C in rapid succession will terminate BENCH.
Benchmark summary information will be written to the listing file for each solver that has successfully completed without any signal interrupt.
In this section we will give a small example showing how to use the BENCH solver and automate the subsequent analysis using the PAVER Server. In particular, we will run the three versions of CONOPT (CONOPT1, CONOPT2, and CONOPT3, as well as CONOPT3 with no scaling) on the default instance of the COPS models for nonlinear programming. We will use the 17 models available from the GAMS Model Library.
Firs we need to extract all of the models from the GAMS Model Library. We can create a file which will extract these automatically. Create a file called
getcops.gms with the entries below:
$call gamslib camshape $call gamslib catmix $call gamslib chain $call gamslib elec $call gamslib flowchan $call gamslib gasoil $call gamslib glider $call gamslib jbearing $call gamslib lnts $call gamslib methanol $call gamslib minsurf $call gamslib pinene $call gamslib polygon $call gamslib popdynm $call gamslib robot $call gamslib rocket $call gamslib torsion
Running the file using
gams getcops.gms extracts the models. Then create a BENCH solver option file called
bench.opt with the entries
solvers conopt1 conopt2 conopt3 conopt3.1 paver 1
The first entry tells BENCH to run the solvers CONOPT1, CONOPT2, and CONOPT3 and then CONOPT3 with the option file
conopt3.opt. The second entry tells BENCH to create PAVER trace files. These can be submitted to the PAVER server for automated performance analysis. Now create an option file
conopt3.opt with the entry
which tells CONOPT3 not to use scaling.
We can now run the models in batch mode, for example by creating a GAMS batch file
runcops.gms with the following entries:
$call gams camshape.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams catmix.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams chain.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams elec.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams flowchan.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams gasoil.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams glider.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams jbearing.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams lnts.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams methanol.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams minsurf.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams pinene.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams polygon.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams popdynm.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams robot.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams rocket.gms nlp=bench optfile=1 reslim=10 domlim=99999 $call gams torsion.gms nlp=bench optfile=1 reslim=10 domlim=99999
Running the file using the command
gams runcops.gms runs all models with all three solvers through the GAMS/BENCH solver. Furthermore, three PAVER trace files are created:
conopt3-1.pvr, where the latter is for CONOPT3 with no scaling. Users can then submit the three trace files to the PAVER Server for automated analysis.
The resulting performance plot in the following figure shows the efficiency of each solver/solver option.