External Equations

Although GAMS provides a powerful language for manipulating data and defining highly structured collections of variables and equations, there are times when you would like to define some parts of your model using a more traditional programming language such as Fortran or C.

This document describes a facility for connecting code written in Fortran, C, Java, Delphi, or some other programming language to equations and variables in a GAMS model. We will refer to these GAMS equations as external equations and the compiled version of the programming routines as the external module defining the external functions (Note that external functions are something completely different than GAMS extrinsic functions, see e.g. a side-by-side comparison). The form of the external module will depend on the operating system being used. The external module under Windows 95/98/NT is a Dynamic Link Library (.dll), and the external module under Unix is a shared object (.so). In principle, any language or system can be used to build the DLL or shared object defining the external module, as long as the interface conventions are not changed.

The basic mechanism is to declare all the equations and variables using the normal GAMS syntax. The interpretation of the external equations is done in a special way. Instead of the usual semantic content, the external equations specify the mapping between the equation and variable names used in GAMS and the function and variable indices used in the external module. This mapping is described in the section on the GAMS interface.

The external module can be written in C, Fortran or most other programming languages. The next sections describe the general definitions for an external module for C, Delphi, and Fortran from a programming language perspective. The way the program is compiled and converted into an external module is system and compiler specific. A set of scripts for various systems and compilers is available together with some examples consisting of GAMS models and C, Delphi, Java, and Fortran files implementing the external functions.

The external equation interface is not intended as a way to bypass some of the very useful model checking done by GAMS when external equations are used with an NLP solver. The external equations a re still assumed to be continuous with accurate and smooth first derivatives. The continuity assumption implies that the external functions must have very low noise levels, considerably below the feasibility tolerance used by the solver. The assumption about accurate derivatives implies that derivatives must be computed more accurately than can be done with standard finite differences. If these assumptions are not satisfied then there is no guarantee that the NLP solver can find a solution that has the mathematical properties of a local optimum, i.e. that satisfies the Karush-Kuhn-Tucker conditions within the standard tolerances used by the solver.

GAMS Interfaces

In order to link an external module to a GAMS model you must specify some mappings. The external functions are assumed to be defined in terms of indices i = 1..m. These indices must be mapped to GAMS equation names. Similarly, the variables used inside the external functions are assumed to be defined in terms of indices j = 1..n. These indices must be mapped to GAMS variable names. Finally, the name of the external module must be specified.

All GAMS solvers are designed for large models and they rely on sparsity in large models. The last part of the specification of a set of external equations is therefore the sparsity pattern of the external functions, i.e. which variables appear in which functions.

The external equations used to specify the interface to the external module are declared in GAMS just like any other equations. For example:

     Equation MyEqn(I, J);

The definition starts out like any other equation, for example:

     MyEqn(I, J)$(Ord(I) ne Ord(J))..

The difference appears after the "..". This part is no longer the algebra for the equation, but rather information that maps each row of the external equation to an index in the external function. An external equation is recognized as external by the equation type "=X=" instead of the usual GAMS equation types "=E=", "=L=" or "=G=". The value of the constant term of the external equation must be an integer constant. The value of the constant maps the row of the GAMS equation to the index (in 1..m) of the external functions. Several blocks of GAMS equations can be mapped to external functions using the =X= notation and external equations can be defined over domains that are restricted by dollar conditions just like any other equations. The mapping between GAMS external equations and external function indices must be one-to-one and the external function indices must be contiguous from one to m (the total number of external functions). This means that two GAMS rows cannot be mapped into the same external function index, and that there can be no holes in the list of external function indices. Although there can be any number of blocks of GAMS external equations, they must all map into and be implemented by one single external function exported by the external module.

The variable part of each external equation defines both the sparsity pattern of the external equation and the mapping from GAMS variable names to the indices of the external variables. The variable part must be a sum of terms where each term is an integer times a variable. The existence of the term indicates that the variable involved is used in the external function and that there is a corresponding derivative. The value of the coefficient defines the index of the external variable (in 1...n) that the GAMS variable is mapped into. For example, the term \(5*Y\) indicates that the external equation depends on GAMS variable Y and that Y is mapped to the 5th element in the vector of external variables. Clearly, if a variable appears in more than one row of an external equation or equations, then the value of its coefficient must be the same in each case.

Several blocks of GAMS variables can be used in the external equations. In contrast to equations, where all rows in an equation block are either external or not, some columns in a variable block can be external while others are not. The mapping between GAMS variables and external variable indices must be one-to-one and the external variable indices must be contiguous from one to n (the total number of external variables). This means that two GAMS columns cannot be mapped into the same external variable index, and that there can be no holes in the list of external variable indices. Although there can be any number of blocks of GAMS variables mapped to external variables, they must all map into one single vector passed to one subroutine in the external module. Inside the module you can of course split the vector of variables any way you like.

Note that while some GAMS variables are external, there is no syntax provided to specify this explicitly, in contrast to the =X= notation use for external equations. External variables can and will be used in normal GAMS equations as well as external equations. Indeed, without this capability the model would be separable and the external equations/functions would be of little use.

The actual functions/equations hidden behind these indexing schemes are always interpreted as equality constraints with zero right hand sides. Inequalities must be converted to equalities by adding explicit slack variables, both in the GAMS model and in the external functions, and the explicit slack must be one of the external variables. A nonzero right hand side has to be taken care of in the external function definition.

The name of the external module in which the external functions are implemented can be defined in a number of ways. By default, the external module is assumed to have the same name as the GAMS model with an extension that is operating systems dependent. The extension is DLL (dynamic Link library) for Windows 95/98/NT and so (shared object) or sl (shared library) for Unix, and it is assumed to be located in the directory from which GAMS is called.

If you would like to use a different name for the external module or if you would like to add a path, use the GAMS FILE statement to define the name of the external file and include the file name as an extra item in the model list. If the file name does not have an extension, a system-dependent extension mentioned above will be added. This is useful when the GAMS model is moved between operating systems. The file name can be used in a model statement, either as an additional item in the list of equation names, or using the syntax /ALL, Filename/.

Since the coefficients and right hand sides in the GAMS definition of the external equations are interpreted as indices, you are not allowed to scale the external equations and variables. Also, since the external equations are treated in a special way, the HOLDFIXED option will not remove any external variables, even if some of them are fixed.

The Programming Interface

This section defines the C, Delphi, and Fortran versions of the GAMS External Function interface. The user must define a function called GEFUNC with the following functionality and interface:

Fortran:

Integer Function gefunc (icntr, x, f, d, msgcb)
C Control Buffer:
Integer icntr(*)
C Numerical Input and Output:
Double Precision x(*), f, d(*)
C Message Callback Routine
External msgcb

C:

GE_API int GE_CALLCONV
gefunc (int *icntr, double *x, double *f, double *d, msgcb_t msgcb)

Delphi:

uses
geheader;
Function GeFunc(var Icntr: ticntr;
var x: tarray;
var F: double;
var D: tarray;
MsgFunc: tMsgCallBack): integer; stdcall;

GE_API is a DLL import/export indicator defined in the header file geheader.h; set -DGE_EXPORTS when building the DLL on a PC to have the DLL export gefunc. GE_CALLCONV indicates the calling convention that should be used. Currently, this is defined to be stdcall on a PC. It is possible to build the DLL using a different calling convention as long as GE_CALLCONV is used for the exported symbol gefunc. There is not such a clean way to specify these things in the Fortran code. When using Watcom Fortran, an auxiliary pragma is used (see geheader.inc), while this can be done via an ATTRIBUTES directive inside the definition of GEFUNC in the case of Visual Fortran.

When building on a Unix machine, both GE_API and GE_CALLCONV are empty.

The icntr vector is a control vector used to pack and communicate control information between the solver and the external module. Some utility functions for handling icntr and the names I_* are defined in the appropriate header file (Fortran: geheader.inc, Delphi: geheader.pas, C: geheader.h). In the following we use square brackets []. Fortran programmers should replace the square brackets with round parenthesis (). The icntr-elements that the implementor of GEFUNC should be concerned with in this initial version of the interface are defined below. The icntr vector is a control vector used to pack and communicate control information between the solver and the external module. Some utility functions for handling icntr and the names I_* are defined in the appropriate header file (Fortran: geheader.inc, Delphi: geheader.pas, C: geheader.h). In the following we use square brackets []. Fortran programmers should replace the square brackets with round parenthesis (). The icntr-elements that the implementor of GEFUNC should be concerned with in this initial version of the interface are defined below.

Element Description
icntr[I_Length]Holds the length of icntr in number of elements. Is defined by the solver and should usually not be used by the implementor.
icntr[I_Neq]Number of external equation rows seen in the GAMS model. Is defined by the solver.
icntr[I_Nvar]Number of external variables seen in the GAMS model. Is defined by the solver.
icntr[I_Nz]Number of nonzero derivatives or Jacobian elements seen in the GAMS model. Is defined by the solver.
icntr[I_Mode]Mode of operation defined by the solver as follows:
1. Initialize. This will be the first call of GEFUNC, where the implementor can perform all initializations needed by the external module.
2. Terminate. This will be the last call of GEFUNC, where the implementor can perform all cleanup tasks needed by the external module.
3.Function evaluation mode. This is the standard mode, where GEFUNC will be called repeatedly during the actual optimization.
Additional mode values may be added later if the functionality of the interface is extended.
icntr[I_Eqno]Index for the external function. icntr[I_Eqno] is defined by the solver, but only in function evaluation mode, i.e. if icntr[I_Mode]=3. icntr[I_Eqno] holds the index of the external function to be evaluated during this call to GEFUNC. icntr[I_Eqno] will be between 1 and icntr[I_Neq], inclusive. Note that the external function interface only allows you to communicate information about one function at a time.
icntr[I_Dofunc]0-1 Flag for function evaluation. Dofunc is defined by the solver, but only in function evaluation mode, i.e. if icntr[I_Mode]=3. When icntr[I_Dofunc] is 1 then GEFUNC must return the numerical value of the function indexed by icntr[I_Eqno] in the scalar f.
icntr[I_Dodrv]0-1 Flag for derivative evaluation. icntr[I_Dodrv] is defined by the solver, but only in function evaluation mode, i.e. if icntr[I_Mode] = 3. When icntr[I_Dodrv] is 1 then GEFUNC must return the numerical value of the derivatives of the function indexed by icntr[I_Eqno] in the vector d.
icntr[I_Newpt]0-1 Flag for new point. icntr[I_Newpt] is defined by the solver, but only in function evaluation mode, i.e. if icntr[I_Mode] = 3. When icntr[I_Newpt] is 1 then the point x will be different from last call of GEFUNC. When icntr[I_Newpt] is 0 then x will not have changed since the last call.
icntr[I_Debug]If icntr[I_Debug] is set to a nonzero value by the implementor then function GEstat and GElog will write all strings to a file called debugext.txt and flush the buffer immediately after writing. The string debugger can be used when a shared object crashes before the solver has had an opportunity to display the messages. In Fortran the string debugger will use Fortran unit icntr[I_Debug].
xEvaluation point. x is a vector with icntr[I_Nvar] elements. It is defined by the solver if GEFUNC is called in function evaluation mode, i.e. if icntr[I_Mode] = 3. The individual elements of x will always be between the bounds defined in the GAMS model. During initialization and termination calls, x will not be defined and you should not reference x. C programmers should index this array starting at zero, i.e. the first external variable is referenced as x[0].
fFunction value: If icntr[I_Mode] = 3 and icntr[I_Dofunc] = 1 then the external module must return the value of external function icntr[I_Eqno] in the scalar f. During initialization and termination calls, f must not be defined.
dDerivative values: If icntr[I_Mode] = 3 and icntr[I_Dodrv] = 1 then the external module must return the values of the derivatives of external function icntr[I_Eqno] with respect to all variables in the vector D. The derivative with respect to variable x[i] is returned in d[i]. It is sufficient to define positions in d that correspond to variables actually appearing in equation icntr[I_Eqno]. Other positions are not being used by the solver and can be left undefined. During initialization and termination calls, d must not be defined.

The msgcb argument to GEFUNC is the address of a message callback routine that can be used to send messages back to the status and log files of the GAMS process. Its type is defined as follows:

typedef void (GE_CALLCONV * msgcb_t)
(const int *mode, const int *nchars, const char *buf, int len);

The mode argument is the logical or of the two flags LOGFILE = 1 and STAFILE = 2 defined in geheader.h and used to direct messages to the log and status files, respectively. nchars is the number of bytes contained in the message (exclusive of the null terminator if there is one present), buf is the address of the message, and len is the size or length of the string buf. The len argument is not used but is included because it makes using this callback from Fortran much easier. GEFUNC must return a status code using the following definition:

Status Code Definition
0 GEFUNC performed as expected.
1 A function evaluation error was encountered. The solver should not use the content of f and/or d, but GEFUNC has recovered from the error and is ready to be called in a new point. This status code should only be used when icntr[I_Mode] = 3.
2 Fatal error. If this value is returned during the initialization call, then the solver should abort immediately. It can be returned by GEFUNC during the initial call if some initializations did not work correctly, or if some of the size values in icntr[.] had unexpected values. It can also be returned during function evaluation mode if the external module has experienced problems from which it cannot recover.

After this description of the appearance of the function GEFUNC, some practical comments about implementing GEFUNC are appropriate.

Compiling and Linking

The set of examples for GAMS External Equations also has a set of scripts for compiling the code on various systems using various compilers. You should use the compiler and linker flags shown in these examples to ensure that the modules conform to the interface standard. You should also use the appropriate include file (C: geheader.h or Fortran: geheader.inc) with compiler directives, preprocessor definitions, and the source code for some utility routines mentioned below.

Initialization Mode

The initialization mode should always check whether the model has the expected size, i.e. icntr[I_Neq], icntr[I_Nvar], and icntr[I_NZ] should be tested against fixed expected values or values derived from some external data set.

The initialization mode can be used for a number of purposes such as allocating memory and initializing numerical information or mapping information needed by the function evaluations to follow. Data can be computed or read from external data sources or it can be derived from calls to an external database. Data that is shared with GAMS can be written to a file from GAMS using the Put statement and read in GEFUNC. Note that you must close the Put file using putclose before the Solve statement. Also note that memory used to hold information from one invocation of GEFUNC to the next should be static. For Fortran it should either be in a Common block or it should be included in a Save statement.

Termination Mode

The termination mode can be used to perform a number of clean-up tasks such as computing statistics, closing files, and returning memory. The solver will terminate shortly after the termination call and a simple implementation may rely of the operating system to close open files and return dynamic memory. However, it is a good habit to perform these tasks explicitly. Sometimes external modules behave different from normal programs.

Evaluation Mode

The bulk of the computational work will usually be in evaluation mode. It is important to recognize that GEFUNC only works with one equation at a time. One of the reasons for this choice is that the addressing of derivatives becomes very simple: there is one derivative for each variable and they have the same index in d and x, respectively.

In some applications several functions are naturally computed together, for example because all functions are computed in some joint integration routine. The Newpt flag is included for these applications. Whenever icntr[I_Newpt] is 1, compute all functions using the common routines, save all function and derivative values, and return the function and/or derivatives values corresponding to equation Eqno. When GEFUNC is called next, icntr[I_Newpt] is most likely 0 and the function and derivative values can be extracted from the information computed (and saved) during the last call where icntr[I_Newpt] was 1.

Evaluation Errors

It is good modeling practice to add bounds to the variables in such a way that all nonlinear functions are defined for all values of the variables within the bounds. Most solvers will also guarantee that nonlinear functions only are called when all elements of the x-vector are between the bounds. However, it may not be practical to add all the necessary bounds, and the implementor of GEFUNC should therefore program the evaluation routine in such a way that division by zero, taking the logarithm of non-positive numbers, overflow in exponentiation and other exceptions are captured. GEFUNC should simply return the value 1 to let the solver know that the function could not be computed at the current point. The solver will not use the function and/or derivative value and it will in most cases be able to backtrack to a safe point and continue the optimization from there.

You should be very careful when using system-default or user-defined function to handle evaluation errors (for example, matherr()). It will not always work in the same way, or at all, inside a DLL or shared object as it does in a self-contained program or a static library.

Communication and Messages

External modules can in general communicate with data sources or other programs by reading and writing files. They can receive information from the GAMS program that called the external module via a Solve statement via PUT files as discussed above. The communication back to GAMS is more limited. Modules can indirectly communicate values back via the solution process, and they can buffer or send messages to the GAMS Status file (usually the .lst file) and the GAMS Log file (usually the screen). Messages to be included in the GAMS Status file can be buffered using the GEstat utility routine described below, and messages to be included in the GAMS Log file can be buffered using the GElog utility routine. Note that you cannot write directly to these files since the solver process will control them.

In addition, you can send messages to both the status and log files, without buffering, using the message callback msgcb. This eliminates the limit imposed by the size of the message buffer and may also make debugging a bit simpler as there is no need to worry about messages that never got flushed from the buffer. Since it may be difficult or impossible to use the message callback from some environments (see the Fortran code provided as an example of this), both the buffered and unbuffered techniques are provided.

N.B. The two techniques for sending messages (buffered via GEstat/GElog and unbuffered via the message callback) are complementary. You can use either one or the other, but if you use both in the same external module, the buffered messages will be printed after the unbuffered ones

GEstat - Utility Routine for writing messages to the Status file

GEstat is provided in the appropriate include file (Fortran: geheader.inc, C: geheader.h). It is used to communicate messages that should be written to the GAMS status file. The function definition is:

Fortran:

subroutine gestat (icntr, Line)
C Control Buffer:
Integer icntr(*)
C input parameters:
character*(*) line

C:

void GEstat (int *icntr, char *line)

Delphi:

Procedure GeStat(var icntr: ticntr; const s: shortstring);

The first argument, icntr, is passed through from the first argument of GEFUNC. The content of 'line' is packed into the control buffer as one line, and when GEFUNC returns, the content of the buffer is written to the GAMS status file. GEstat can be called several times, each time with one line. 'Line' should not be longer than 132 characters and the overall amount of information written in one call to GEFUNC should not exceed 1000 characters. 'Line' should not contain any special characters such as new-line, tab, or null characters.

In practice, GEstat is often used with calls like the following:

Fortran:

call GESTAT (icntr, ' ')
call GESTAT (icntr, '**** External module based on abc.for')

C:

GEstat (icntr, " ")
GEstat (icntr, "**** External module based on abc.c")

Delphi:

gestat(icntr,' ');
gestat(icntr,'**** External module based on abc.dpr');

It should be mentioned that you cannot write directly to the GAMS status file since this file will be opened and controlled by the Solver process.

GElog - Utility Routine for writing messages to the Log file

GElog is also provided in the appropriate include file. It is used to communicate messages that should be written to the GAMS Log file. The Log file is usually the screen, but it can also be a file, see the GAMS LF parameter. The function definition of GElog is:

Fortran:

subroutine gelog( Icntr, Line )
C Control Buffer:
Integer icntr(*)
C input parameters:
character*(*) line

C:

void GElog(int *icntr, char *line)

Delphi:

Procedure GeLog (var icntr: ticntr; const s: shortstring);

The content of 'line' is written to a buffer that in turn is written to the log file when GEFUNC returns. GElog behaves exactly like GEstat, with Status file replaced by Log file.

It should be mentioned that you cannot write directly to the screen with some combinations of operating system and compiler; this may also depend on the options or flags you use to build the external module. On some systems this may cause your external module to crash. So, do not use writing to the screen as a method for debugging unless you know it works. Otherwise you may continue to crash because of the debugging statements after all other errors have been removed. Write to a file and flush the buffer as a safe alternative.

The Message Callback MSGCB

Calling the message callback msgcb from C or C++ is relatively straightforward. You should note that the mode argument, the nchars argument, and the buf argument are all call-by-reference and that addresses, not values, must be used. The len argument is the exception and is call-by-value; you should pass the value *nchars. If you are implementing in Delphi or VB, you should note that pointers of all types are 4-byte quantities, as are ints.

Calling this routine from a Fortran environment is a bit more complicated due to the different ways that Fortran compilers handle strings. The Unix convention (at least the one observed on all systems for which GAMS is built) is that strings are passed by reference, but that in addition, the length of the string is passed, by value, as a hidden 4-byte quantity appended to the end of the argment list. This is the reason for the len argument at the end of the argument list for msgcb. This final argument makes it possible to make Fortran callbacks in a Unix environment as

character*(*) msgbuf
int nchars, charcount
nchars = charcount(msgbuf)
call MSGCB (mode, nchars, msgbuf)

Similar reasoning applies to the Visual Fortran compiler on the Intel platform, where instead of being placed at the end of the argument list, the hidden length arguments are mixed into the argument list. Each character address is followed immediately by the length (again, passed by value) of the character string. Since the only string passed in the example above is at the end of the argument list, the code will work for Visual Fortran on Intel.

In the case of the Watcom Fortran compiler, the default mechanism for passing strings makes use of the string descriptor (a structure consisting of a length and an address). The address of this string descriptor is what is passed. This makes it necessary to use a more elaborate scheme to use the message callback from Watcom Fortran. In our examples, we illustrate two techniques for handling this.

In the first example (see the files ex1f_cb.for and watmsg_c.c), we simply make a Fortran call to a C wrapper. The C wrapper can use the address of the string descriptor passed to it from the Fortran routine to access the message buffer, and uses this to call the message callback routine. In this case, we don't do any tricks or contortions in the Fortran code, but instead write C code to accomodate the Fortran calling convention. Once we have the string in a C environment, it is clear how the callback should be used.

In the second example (see the files ex1f_cb.for and watmsg_f.for), we don't introduce an additional layer with the C wrapper. Instead, we use an auxiliary pragma to adjust the calling conventions used to make the call to msgcb from the Fortran code. The auxiliary pragma forces the Watcom Fortran compiler to use the stdcall calling convention (this affects how the parameters are passed). In addition, the pragma specifies, via a parm list, what is passed for each argument of the msgcb function. In this case, we specify that the first two arguments (mode and nchars) be passed by reference (the default), the message string be passed by data_reference (only a pointer is passed), and the final len argument be passed by value.

Communicating data to the external module via files

Some external functions will need data from the GAMS program. These data can be passed on via one or more files written using PUT statements. Usually, the PUT files will be written in the current directory and an OPEN statement in the external function will look for them in the current directory. However, if you need to run multiple copies of the same model at the same time, you should write the data file in the GAMS scratch directory and direct the external function to look for it in the scratch directory.

In the GAMS model you define the put-file to be in the scratch directory with a statement like

FILE f / '%gams.scrdir%filename' /;

You should use the extension dat to ensure that GAMS will remove the file cleanly when it finishes. If another extension is used and you don't delete the file, GAMS will complain about an unexpected file when it cleans up after the run. The external function can get the name of the scratch directory from the solver during initialization. You set Icntr(I_Getfil) = I_scr and return immediately. The solver will store the name and length of the scratch directory in the communication buffer, and call the external function again, this time with a sub-mode, Icntr(I_Smode) set to I_Scr. You can then extract the name using the Fortran call

call GENAME( Icntr, Scrlen, Scrdir )

where Scrdir (declared as character*255) will receive the scratch directory, and Scrlen (declared as integer) will receive the actual length of Scrdir. In C, the call looks like

scratchDirLen = GEname (icntr, scratchDir, sizeof(scratchDir)); 

where the routine returns -1 on error, and the number or characters transferred to the buffer scratchDir on success. If there is room, a terminating null byte is written to scratchDir. If the value returned is equal to sizeof(scratchDir), then the string returned will not be null-terminated and may have been truncated as well.

You can get other directory or file names by setting Icntr(I_Getfil) to the following values:

Value Description
I_ScrValue used to request the scratch directory
I_WrkValue used to request the working directory
I_SysValue used to request the systems directory
I_CntrValue used to request the control file

After setting the Getfil flag you must always return immediately, and ncall, and the relevant call will be flagged with the sub-mode in Icntr(I_Smode).

The models ex5.gms and exmcp3.gms with their corresponding Fortran and C source files provide an example.

Constant Derivatives

Some solvers (the CONOPT solvers in particular) can take advantage of constant derivatives corresponding to linear terms. This can be especially useful if an external function represents an equation like Y = expression(X), where Y is unbounded and only appears in the objective function.

In the external module interface described above the solver cannot see that Y appears linearly. An optional extension allows advanced users to indicate that some of the relationships are linear. During the setup call, the user must define

Icntr[I_ConstDeriv] = Number of constant derivatives 

If the solver can use this information (not all solvers will) then GEFUNC will be called again with I_Mode = 4 (or symbolic constant DOCONSTDERIV in C), once for each equation defined in the usual way in I_Eqno. The user must for each of these calls define the values of the constant derivatives in the D-vector. The remaining elements of D, both those corresponding to varying derivatives and to zeros, must be left untouched. These special calls will take place after the setup call and before the first function evaluation call. In these calls other arguments such as I_Dofunc, I_Dodrv, and X will not be defined.

The model ex4x with the corresponding Fortran and C source files ex4xf_cb.for and ex4xc_cb.c shows an example of how to use constant derivatives. The files can be compared to the corresponding files ex4f_cb.for and ex4c_cb.c without constant derivatives.

Second Derivatives: Hessian times Vector

Some Solvers (initially CONOPT3) can take advantage of second order information in a special form: the Hessian matrix times a vector. If this information can be supplied (for all external equations) then the user must during the setup call define

Icntr[I_HVprod] = 1

If the solver can use this information (not all solvers will) then GEFUNC may be called with I_Mode = 5 to request this information. I_Eqno will hold the equation number, and X will in its first I_NVar positions hold the values of the variables, and in the next I_Nvar positions will hold a vector V. The user should evaluate and return D = H*V for the particular equation in the particular point. D will be initialized to zero by the solver.

H*V will often be needed for several vectors V in the same point X. I_Newpt will be used to indicate changes in X in the usual way.

The model ex1x with the corresponding Fortran source file shows how to use both constant derivatives and second derivatives.

Examples

There are several examples in the GAMS Test Library. The model [testexeq] gives an overview of the examples available and can be used to compile and run these examples and a selection of those respectively. The Test Library model [complink] can be used as a script to compile and link external equation libraries.

Regardless of how you build the external libraries, the different examples, e.g. [ex1] , will by default solve one model using no external equations. To solve it with all kinds of different external equation libraries you can run the model with "--runall=1". Alternatively, only selcted libraries will be used by setting one or more of the following flags: "--runC=1", "--runC_cb=1", "--runD=1", "--runD_cb=1", "--runF=1", "--runF_cb=1", "--runJ=1".

Debugging External Equations

External Equations combine different computer languages that are not normally capable of communicating. In particular, these languages cannot communicate about argument conventions, execution errors, and exceptions. It is therefore not possible to provide the same level of protection and reliability as with pure GAMS models.

There are many potential sources of errors. The argument lists in the C or Fortran code can be incorrect, or the linking process can create an incorrect external module. There is little GAMS can do to help you with this type of errors. You must carefully follow the examples and apply testoutput during the setup calls, for example using GEStat and GElog.

Once the overall setup is correct and GAMS can establish proper communication with the external module there is still a chance of numerical errors where the function values and the derivatives do not match. The CONOPT3 solver will be default call its 'Function and Derivative Debugger' in the initial point if a model has any external equations. The debugger will check that the functions only depend on the variables that are defined in the sparsety pattern and that derivatives computed by numerical perturbation are consistent with the derivatives computed by the external module. If an error is found CONOPT3 will stop immediately with an appropriate message.

The examples er1 to er3 have different types of errors, and you can see the coresponding error messages if you use CONOPT3 as the NLP solver. Comments about the errors can be found in the C or Fortran source code.

Please note that several types of errors cannot be found. Derivatives computed in the external module and returned in positions that were not defined in the sparsety pattern in GAMS are filtered out by the interface and are therefore not found. Similarly, derivatives that should be computed but are forgotten may inherit values from the same derivative in another equation computed earlier. Finally, we cannot perturb fixed variables so errors related to these variables will usually not be detected.

There are four CONOPT3 options that control the Function and Derivative debugger. These options can be changed in the usual way in a CONOPT3 options file:

Option Description
LkdebgDebug control: Controls how often the Function and Derivative debugger is called. The default value is -1 which means that is is called in the initial point. The value 0 will turn the debugger off, and the value +n means that the debugger is called every n'th time derivatives are computed.
LfderrOutput Control: Controls the number of error messages. Only the first Lfderr errors will appear in the GAMS list file. The default value of Lfderr is 10.
Rtmxj22nd Derivative Noise: A derivative is consider incorrect if the derivative computed by the external module deviates from the numerically computed derivative by more than Rtmxj2 * step. If the derivative is correct, this can only happen if the 2nd derivative is greater than Rtmxj2. The default value of Rtmxj2 is 1.e4 and it can be increased if necessary.
RtzernZero Noise: If external functions have constant derivatives then the constant terms are still part of the external function and this can give rise to small inaccuracies in the contribution of the constant derivatives to the function value. This noise can cause the debugger to incorrectly state that a the external equation depend on the variable with the constant derivative. A larger value of Rtzern may cure the problem. The default value is 1.e-14.