Table of Contents
Interfacing GAMS and MATLAB. This document briefly describes GDXMRW (GDX-Matlab Read/Write), a suite of utilities to exchange data between GAMS and MATLAB. The software gives MATLAB users the ability to use all the optimization capabilities of GAMS, and allows visualization of GAMS models directly within MATLAB. The most recent version of GDXMRW is included as part of the current GAMS Distribution.
- Date
- August 1, 2014
- Attention
- GDXMRW is deprecated (see GAMS 38 GDXMRW release notes). Please use GAMS Transfer Matlab instead.
Introduction
Optimization is becoming widely used in many application areas as can be evidenced by its appearance in software packages such as Excel and MATLAB. While the optimization tools in these packages are useful for small-scale nonlinear models (and to some extent for large linear models), the lack of a capability to compute automatic derivatives makes them impractical for large scale nonlinear optimization. In sharp contrast, modeling languages such as GAMS and AMPL have had such a capability for many years, and have been used in many practical large scale nonlinear applications.
On the other hand, while modeling languages have some capabilities for data manipulation and visualization (e.g., Rutherford's GNUPLOT), specialized software tools like Excel and MATLAB are much better at these tasks.
This paper describes a link between GAMS and MATLAB. The aim of this link is two-fold. Firstly, it is intended to provide MATLAB users with a sophisticated nonlinear optimization capability. Secondly, the visualization tools of MATLAB are made available to a GAMS modeler in an easy and extendable manner so that optimization results can be viewed using any of the wide variety of plots and imaging capabilities that exist in MATLAB.
In order to enable this link between GAMS and MATLAB, we have implemented MATLAB callable functions that can efficiently import and export data to and from GAMS through GDX files. The simplest read and write functions, irgdx
and iwgdx
, deal with indexed parameters. Without getting into their precise meaning, these parameters are essentially indexed in each dimension by the sequence of integers from 1 to the size of that dimension. As an example of the read function, the following command in MATLAB will store the matrices A
and B
into the caller workspace after reading from the file foo1.gdx
:
>> irgdx('foo1','A','B');
Similarly, the call
>> iwgdx('foo2','C','D');
would write the matrices C
and D
(located in the MATLAB workspace) as indexed parameters into a file named foo2.gdx
. Note that, many of the examples found in this manual are also part of the GAMS data utilities models library, referred to here as datalib. The statements above can be found in datalib example gdxmrw_intro02_init
. For reading and writing more complex data such as sets, variables, equations, and non-indexed parameters, we can use the more sophisticated functions rgdx
and wdgx
. Further descriptions of these functions together with advanced use of irgdx
and iwgdx
are detailed in the rest of this paper.
In Section Data Transfer, we discuss the data transfer utilities that allow importing and exporting data between MATLAB and GDX files: irgdx
, iwgdx
, rgdx
, wgdx
. In Section Extended use, we give a few examples of the MATLAB and GAMS interface. In APPENDIX A - Configuring GDXMRW, we provide information about configuring GDXMRW and testing the utilities. In APPENDIX B - Utility functions: gdxWhos and gdxInfo, we describe additional utility functions, gdxWhos
and gdxInfo
, that allow viewing contents of GDX files in the MATLAB console. Finally, in APPENDIX C - Calling GAMS model from MATLAB, we discuss the gams
function that with a single call initializes a GAMS model with MATLAB data, executes GAMS on the model, and returns results back into MATLAB.
Data Transfer
This paper describes a suite of tools for exchanging data between GAMS and MATLAB. This data exchange is accomplished via the GDX (i.e. the GAMS Data eXchange) interface and API. There are many advantages to using GDX, including platform independence, space and time efficiency in storing and accessing data, and a guarantee that all GDX data contains no duplicates and is free from any logical or syntax errors that might prevent it from being read into GAMS. The GDX interface is well tested, available to the public, and is the basis for most if not all of the GAMS data utilities.
In this section we discuss four MATLAB routines. The first two, irgdx
and iwgdx
, are the subject of sections irgdx and iwgdx and are used to quickly and simply read indexed parameters from a GDX file into MATLAB and vice versa. For an example use of these functions, a generic quadratic program function that mimics MATLAB's quadprog
function is provided in datalib example gdxmrw_qp3
. The rgdx
and wgdx
routines (sections rgdx and wgdx) are more sophisticated versions of the first two that can read and write more general GDX data, e.g. sets, variables, equations, and non-indexed parameters. To understand the structure of this GDX data, the material in section Indexing with labels (UELs) is essential.
irgdx
irgdx
is a specialized MATLAB function to do an efficient import of an indexed parameter from a GDX file. The irgdx
call will not read non-indexed parameters; it can only read data recognized as an indexed data type. If a GDX file consists of both indexed and non-indexed data, irgdx
can still read the indexed data. To see whether particular parameters in a GDX file are indexed, the gdxWhos
function described in APPENDIX B - Utility functions: gdxWhos and gdxInfo can be used.
The syntax and functionality of irgdx
closely resembles that of the MATLAB load
function to import MATLAB formatted data, i.e data stored in MAT files. In Basic syntax and Load symbols into output structure we describe the basic modes of operation where the data read from GDX are either stored in the caller workspace or into a MATLAB structure on the left-hand side of an assignment. In MATLAB sparse form, we describe an optional specification that allows read results to be stored in MATLAB sparse form. Finally, in Subsection Renaming, we describe a simple syntax that allows the renaming of variables/parameters without the need to make expensive copies.
Basic syntax
The basic syntax for irgdx
is
irgdx('gdxFileName', 'sym1', 'sym2', ...);
The above call will read the indexed parameters named sym1, sym2, and so on from the GDX file whose name is specified in the first argument and store the results in the MATLAB caller workspace. Only the GDX name argument is mandatory. If only this argument is specified (i.e. no symbol names are provided) then all of the indexed parameters stored in the GDX file will be loaded into the workspace. The GDX name can be specified with or without the '.gdx'
extension. Note that all of the input arguments must be specified in string form.
As an example of the above syntax, we can read the scalar a0, the vector a1, and the matrix a2 from the file idx1_.gdx
with the following command:
>> irgdx('idx1_','a0','a1','a2');
The three symbols are stored as MATLAB variables a0, a1, and a2 in the caller workspace. If we want to read all the symbols from the file idx1_.gdx
, we simply call irgdx
with only the file name:
>> irgdx('idx1_');
Load symbols into output structure
An alternative to storing the results in the MATLAB caller workspace is to return the results in a MATLAB structure. Here, the syntax is:
s = irgdx('gdxFileName','sym1','sym2',...);
This call will return the indexed parameters named sym1
, sym2
, and so on in the structure s, with fields s.sym1
, s.sym2
, and so on containing the values of the respective indexed parameters. The input arguments must all be in string form. Only the GDX name is mandatory; if only this argument is specified, all of the indexed parameters stored in the GDX file will be returned in the output structure. The GDX name can be specified with or without the .gdx
extension.
For example, the following call returns the values of parameter a2 in the structure s:
>> s = irgdx('idx1_','a2'); >> s.a2 ans = 50 50 50 50
To return all the indexed parameters in the fields of the structure s
, we would simply call irgdx
with only the GDX name:
>> s = irgdx('idx1_');
See datalib example gdxmrw_irgdx01_init
.
MATLAB sparse form
By default, irgdx
will store results in MATLAB dense form. However, it is possible to use MATLAB sparse form for parameters with dimension less than or equal to 2; MATLAB's sparse storage scheme cannot handle matrices with more than 2 dimensions. Using sparse storage will allow for efficient import of especially large sparse indexed parameters. Note that in this context sparse does not refer to data represented in the form [i,j,...,val] but rather to MATLAB's internal sparse storage scheme. To store a particular symbol in MATLAB sparse form, one can simply append ':s' to that symbol's name in the irgdx
call. For example, the following call will store the parameter a2 into the caller's workspace in sparse form:
>> irgdx('idx1_','a2:s');
Note that if we want to store all the indexed parameters from a specified file in MATLAB sparse form, we would need to specify each symbol individually in order to append the ':s' to each symbol's name.
Renaming
The details of this section are rather involved; it can be skipped without loss of continuity.
While an irgdx
call will typically store results in the caller workspace or create field names in the output structure using the symbol names from the GDX file, it is possible to rename the outputs in MATLAB without making potentially expensive copies. To rename a symbol, we can take advantage of MATLAB's default copy-on-write mechanism: when an assignment occurs, the values assigned are not actually copied until one of the values (the original or the copy) is actually changed. For example, after executing the following two lines of code, the variable a2_new
points to the same memory location as a2 and is not a separate copy of a2:
>> irgdx('idx1_','a2'); >> a2_new = a2;
If however the value of a2_new
were changed after executing the above lines, MATLAB would then need to make a separate copy, at the cost of memory and time. Now, the command
>> clear a2;
immediately following the above code would complete the rename of a2
to a2_new
as desired. Once a2
is cleared, we can change a2_new
without needing to make a copy. Similarly, we can do a rename when we have an output structure:
>> s = irgdx('idx1_','a2'); >> s.a2_new = s.a2; >> s = rmfield(s,'a2');
To convince ourselves that this code indeed does not require a copy, we can run the commands in the debug output format. That is, immediately after the assignment of s.a2
to s.a2_new
in the above code (before using the rmfield function), we can run the following commands:
>> format debug; >> s.a2 ans = Structure address = 8c1eda0 m = 2 n = 2 pr = 75fa88a0 pi = 0 50 50 50 50 >> s.a2_new ans = Structure address = 8c1eda0 m = 2 n = 2 pr = 75fa88a0 pi = 0 50 50 50 50
Note that the pointer pr is the same for both outputs, implying that MATLAB does not make a separate copy prior to a write. A similar test can also be done when parameters are stored in workspace variables rather than a structure.
iwgdx
iwgdx
is a function that creates a GDX file of indexed parameters from MATLAB data. Since iwgdx
writes only indexed parameters to GDX, the input MATLAB data is simplified: no labels are required or implied. The input matrices can be stored in the usual (dense) scheme or using the MATLAB sparse scheme: iwgdx
detects and handles the two cases automatically. Note that matrices represented in [i,j,...,val]
form are not acceptable as iwgdx
input.
While the iwgdx
function is patterned after the save
function in MATLAB, there are important differences. Firstly, if only
the file name is specified, the save
function stores all variables in the caller workspace, while iwgdx
creates an empty GDX file. The behavior of iwgdx
in this case also differs from that of irgdx
, which reads all parameters when only the file name is specified. Secondly, iwgdx
has an additional pass by reference and renaming syntax, in which the parameter string name and values are passed as two consecutive arguments. In Subsection Basic syntax, we describe the basic syntax for iwgdx
. In Subsection Pass by reference syntax , we introduce the pass by reference and renaming syntax. Finally, in Subsection Dimensionality specification, we describe an optional dimensionality specification that allows symbols to be written with a larger dimensionality than the default.
Basic syntax
The basic syntax for iwgdx
is:
iwgdx('fileName','sym1','sym2',...);
The above call will write the values of the variables sym1, sym2, and so on in the MATLAB caller workspace into a GDX file with the name given by the first input argument. The file name can be specified with or without the '.gdx' extension. If a GDX file with the same name already exists, it will be overwritten. Note that a call with a single input argument (the GDX name) will raise a warning and will create an empty GDX file. This contrasts to the MATLAB save function, which writes all of the workspace variables when none are specified. In the basic form described here, all of the iwgdx
input arguments are strings.
To illustrate, suppose the matrices a2 and a3 exist in the MATLAB caller workspace. We can write the results of these matrices into a file test.gdx with the following call:
>> iwgdx('test','a2','a3');
If the file test.gdx already exists, it will be overwritten. Specifying only the file name will create an empty GDX file:
>> iwgdx('test'); Warning: an empty gdx file was created.
Pass by reference syntax
When using the basic iwgdx
syntax, a copy of the workspace variable to be written to GDX is made as part of the call. This is unavoidable: the MATLAB executable (MEX) interface only provides functions that return copies of workspace variables, most likely to avoid overwriting or corrupting data. However, an alternative, more efficient approach is possible by passing the numeric data as an argument. Essentially, the pointer to the data values are passed by reference and no copy is made. This could be especially useful when writing a large amount of data. An additional benefit of this syntax is the ability to write GDX parameters with different names than used in the MATLAB caller workspace. For example:
>> iwgdx('test','s1',s1Mat,'s2',s2Mat);
In the above command, s1
and s2
are strings holding the GDX parameter names, while s1Mat
and s2Mat
are matrices containing the values to store. We can also use a mix of the basic syntax with the pass by reference syntax. For example:
>> iwgdx('test','anew',a,'b','c',c);
This call would save matrix a
as anew in GDX without making a copy, would internally make a copy of b
while saving b
to GDX, and would save c
to GDX without renaming or making a copy.
We can do a timing test to see the performance difference between the basic syntax and the pass by reference syntax. The following commands create a 5000x5000 dense matrix consisting of random values in the workspace.
>> clear; >> randMat = rand(5000);
First, we do a timing test using the basic syntax:
>> delete test.gdx; % delete file if it exists >> tic; >> iwgdx('test','randMat'); >> t1 = toc t1 = 5.3346
Now, we do a timing test using the pass by reference syntax:
>> delete test.gdx; % delete file if it exists >> tic; >> iwgdx('test','randMat',randMat); >> t2 = toc t2 = 5.1636
For the above example, the improvement in time performance is about 3%. Of course, the pass by reference call will also use only the half the memory compared to the basic call. We can expect further improvement in performance time and memory use for even larger data.
Dimensionality specification
It is possible to specify the number of dimensions (i.e. dimensionality) for the symbol written to GDX by appending :n
to the variable/symbol name, where n
is the desired dimensionality of the GDX symbol. This feature is motivated by two quirks of the MATLAB environment. Firstly, all scalars and vectors are stored in MATLAB as 2-dimensional matrices. For example:
>> s0 = 100 >> s1 = [ 100 ]
results in two variables with identical storage schemes: 1x1 matrices. In the MATLAB environment this isn't a problem: things behave as you would expect them to, and the isscalar
, isvector
, and ismatrix
calls are available to interrogate MATLAB about how it views variables. However, in GDX there is a distinction between a scalar (something with 0 dimensions), a parameter with 1 dimension and length 1, and a 1x1 parameter. We need to be able to create GDX files containing any or all of these.
Secondly, MATLAB "flattens" all variables by removing trailing dimensions whose size is 1. For example, after executing
>> B = ones(2,2,2) >> C = ones(2,2,2,1,1)
B
and C
will be identical 3-dimensional MATLAB arrays, yet we need to be able to create a 2x2x2x1x1 indexed parameter in GDX.
By default, iwgdx
creates GDX parameters that are consistent with the MATLAB view of the data passed in. For example, executing
>> d0 = ones(1,1) >> d1 = ones(3,1) >> d2 = ones(3,3) >> iwgdx('dd','d0','d1','d2')
results in indexed parameters with 0, 1, and 2 dimensions, respectively, being written to GDX. However, if we want to write d0 as a vector or matrix or higher-dimensional array, we need to use the optional syntax to specify the dimensionality of the resulting symbol in GDX. For example:
>> d0 = pi; >> p1 = d0; >> d1 = pi*ones(3,1); >> p2 = d1; >> d2 = pi*ones(3,3,1) >> iwgdx('pp','d0','p1:1','d1','p2:2','d2','p3:3',d2)
will write the following symbols to GDX:
- The 0-dimensional symbol
d0
. This is the default behavior, consistent with the MATLAB view thatd0
is a scalar. - The 1-dimensional symbol
p1
. The scalar variablep1
was promoted to a 1-dimensional symbol by adding an additional singleton dimension. - The 1-dimensional symbol
d1
. This is the default behavior, consistent with the MATLAB view thatd1
is a vector. - The 2-dimensional symbol
p2
. The vector variablep2
was promoted to a 2-dimensional symbol by adding an additional singleton dimension. - The 2-dimensional symbol
d2
. This is the default behavior, consistent with the MATLAB view thatd2
is a matrix, i.e. the final singleton dimension is just removed. - The 3-dimensional symbol
p3
. The matrix variabled2
was promoted to a 3-dimensional symbol by adding an additional singleton dimension. Independently, call-by-reference was used to renamed2
top3
.
Note that the dimensionality specified must always be a promotion, i.e. the dimensionality of the resulting GDX symbol is only increased from what it would be by default.
See datalib example gdxmrw_iwgdx01_init
.
Indexing with labels (UELs)
To this point we have been looking at functions to read (irgdx
, Section irgdx) and write (iwgdx
, Section iwgdx) indexed parameters from and to GDX. Indexed parameters are convenient since their structure is essentially identical to that of MATLAB matrices. This structure can be encapsulated very simply as the number of dimensions and the size or extent of each dimension. Using indexed parameters, we do not need to be very concerned with the structure of the data since little structure exists. However, there is much more to GDX data than indexed parameters. If we want to read and write GDX data in more generality we will need to understand how labels or strings are used to reference GDX data and how these labels are organized within GDX.
In general, GAMS data is referenced with labels instead of with numbers, so that one references demand('chicago')
instead of demand(2)
in a GAMS model. These labels are also called Unique Elements, or UELs. The collection of UELs used in a model or in a GDX file is ordered internally and often referred to as the universe of UELs. For efficiency, it is not necessary to use labels internally in GDX or when using the GDX API. Instead, a correspondence or mapping between integers and labels is established initially and/or built up as labels are introduced, so that integers can be used in place of the UELs that reference the data. It is important to keep in mind that the labels are the key thing in referencing GDX data: when integer maps are used the integers are only used to efficiently represent the labels.
A similar scheme mapping integers to labels is used in the rgdx
and wgdx
routines for reading and writing general GDX data. In addition to the actual data values being stored, there will be a mapping passed to allow integers to be mapped to labels and vice versa. In some cases, where no mapping is passed and data are written to GDX, a default mapping (1
to '1'
, 2
to '2'
, etc.) may be used. When reading, the default mapping to use when passing back data is the universe mapping from the GDX, but it is possible to apply a filter. A filter can reduce the amount of data returned and also change the order of that data.
rgdx
rgdx
is a MATLAB utility to import data from a GDX file. It takes structural input and returns data back in the form of a structure. This is a very flexible routine as it gives user control over the output data structure. rgdx
can read a set/parameter/equation/variable from a GDX file and display results in either full/dense or sparse form. A user can also perform a filtered read to read only certain specific elements of a symbol. It can also perform compression to remove extra zeros.
This routine can take up to two arguments. The first argument is a string input containing the GDX file name. It can be with or without the '.gdx'
file extension. If you call this routine with only the GDX file name as an argument then the 'uels'
field of output structure will be the global UEL of the GDX file and the rest of the fields of the output structure will be NULL. The second argument is a structure input containing information regarding the desired symbol. The syntax for this call looks like this:
x = rgdx('fileName', structure);
As an example, we read a 3D parameter, 'test3'
from 'sample.gdx'
. Here we display this parameter in full format but without redundant zeros:
>> s.name = 'test3'; >> s.form = 'full'; >> s.compress = true; >> x = rgdx('sample', s) x = name: 'test3' type: 'parameter' dim: 3 val: [4x2x2 double] form: 'full' uels: {{1x4 cell} {1x2 cell} {1x2 cell}} >> x.val ans(:,:,1) = 3 4 4 5 5 6 6 7 ans(:,:,2) = 4 5 5 6 6 7 7 8 >> x.uels{1} ans = '1' '2' '3' '4' >> x.uels{2} ans = 'j1' 'j2' >> x.uels{3} ans = 'k1' 'k2'
In the following subsections we will explain the input and output structures. Please note that except for the 'name'
and 'uels'
fields, all other string fields take case insensitive input. All boolean fields can also be entered as string values as well.
Input structure
To read a symbol from a GDX file we just need to know its name in string format. Thus, the only mandatory field of the input structure is 'name'. e.g.
>> s.name = 'test3';
There are several other optional fields of the input structure that give user more control over the output structure. These optional fields are as follows:
- form
This field represents the form of the output data. Output data can be either in'full'
or'dense'
form or it can be in[i, j,.., val]
sparse form. We will label[i, j,.., val]
as 'sparse'. A user can enter it as string input with value'full'
or'sparse'
. e.g.>> s.form = 'full';
By default the data will be in 'sparse' format. Note that this sparse format differs from MATLAB's internal sparse storage scheme that we referred to in prior sections for theirgdx
andiwgdx
functions. - compress
By default the uels in the output structure will be a global UEL of the GDX file and the'val'
field data will be indexed to this UEL. Thergdx
routine allows a user to remove rows and columns with all zeros from the 'val' data matrix and re-indexes the uels accordingly. This is called compression of the data. This can be achieved by setting compress as true in the input structure. Valid values for this field are true and false, either in logical form or in string form. e.g.>> s.compress = 'true';
However, we note that compressing the data can be dangerous because the size of the matrix that is read can be incorrect. Essentially, all zero rows and columns are removed, including those that might actually be part of the data values in the symbol matrix. - uels
This input field is used to perform a filtered read, i.e. output data matrix will contain values only corresponding to the entered uels. Filtered read is very useful if user just wants certain specific set of data. Uels should be entered in cell array form. It has to be in 1xN form with each column being a cell array representing the uels for that dimension. Each column can have strings, doubles or combinations of both. It also allows a user to enter double data in shorthand notation or a 1 x N matrix. For example, in the previous example we can perform a filtered read to get data corresponding to only the'1'
,'3'
elements of the first index of the parameter'test3'
. The following function is handy when one needs to generate a UEL listing for the input structure:>> guel = @(s,v) strcat(s,strsplit(num2str(v)));
Using the above function, we can create a listing of strings consisting of the string s appended to each number in the array v. Thus, instead of using the command>> s.uels = {{'1','3'},{'j1','j2'},{'k1','k2'}};
we can conveniently use the command>> s.uels = {guel('',[1,3]),guel('j',1:2),guel('k',1:2)};
The benefit of using the latter command will be more apparent when creating many elements in the listing. Now, as an example, suppose we would like to do a filtered read on the parameter calledtest2
insample.gdx
. We could use the following commands:>> s.name = 'test3'; >> s.form = 'full'; >> s.compress = false; >> s.uels = {guel('',[1,3]),guel('j',1:2),guel('k',1:2)}; >> x = rgdx('sample',s) x = name: 'test3' type: 'parameter' dim: 3 val: [2x2x2 double] form: 'full' uels: {{1x2 cell} {1x2 cell} {1x2 cell}} >> x.val ans(:,:,1) = 3 4 5 6 ans(:,:,2) = 4 5 6 7
Here it should be noted that we turned off compression while performing the filtered read. This is necessary because the filtered read will give data in accordance with the entered uels and the output uels will be the same as the input uels; thus compression is not possible. - field
This field is required when variables or equations are to be read from a GDX file. Sets and parameters in the GDX file do not have any field value but variables and equations have 5 fields namely, level, marginal, lower, upper, and scale. Thus, it may be useful to enter field as an input when reading an equation or a variable. A user can enter it as a string with valid values being'l/m/up/lo/s'
. e.g.>> s.field = 'm';
By default, the output will be the level value of a variable or an equation. - ts
This represents the text string associated with the symbol in the GDX file. If a user sets this field to be'true'
, then the output structure will have one more string field'ts'
that contains the text string of the symbol. e.g.>> s.ts = true;
- te
GAMS allows a modeler to enter text elements for a set. Similarly to the'ts'
field, if a user sets'te'
to be true in the input structure, then the output structure will contain one more field representing the text elements for that symbol. Please note that text elements only exist for'sets'
. e.g.>> s.te = true;
Output Structure
As mentioned earlier, output of the rgdx
routine will be in structure form. This structure is very similar to the input structure. To get information regarding any symbol, we always need to display its basic characteristics, such as its name, type, value, uels, form, etc. An output structure will always have these fields:
- name
It is same as that entered in the input structure name field, i.e., the symbol name in the GDX file. - val
It represents the the value matrix of the symbol. To save MATLAB memory by default it will be in'sparse'
format. e.g.>> s = rmfield(s, 'form'); >> s s = name: 'test3' compress: 0 >> x = rgdx('sample', s) x = name: 'test3' type: 'parameter' dim: 3 val: [16x4 double] form: 'sparse' uels: {{1x8 cell} {1x8 cell} {1x8 cell}}
Hereval
is a 16x4 double matrix. As it is a parameter; thus the last column of the sparse matrix will represent the value and the rest (i.e. the first three columns) will represent its index. Please note that in the case of a'set'
, the number of columns in the sparse matrix will be equal to its dimension, i.e., it will not have a column representing its value. Here, the presence of each row in the output'val'
field corresponds to the existence of a set element at that index. When a'full'
matrix output is specified, a 1 represents existence of a set element and a 0 otherwise. - form
It represents the format in which the'val'
field is being displayed. As mentioned earlier it can be either in'full'
or'sparse'
form. - type
While reading a symbol from a GDX file it is often very useful to know its type. Thergdx
routine is designed to read set, parameter, variable and equation. This field will store this information as a string. - uels
This represents the unique element listing of the requested symbol in the form of a cell array. It is a 1 x N cell array, where N is the dimension of the symbol. Each column of this array consists of string elements. By default, the output uels will be the same as the global uel of the GDX file, but it can be reduced to element specific local uels if compress is set to be true in the input structure. If a user is using a filtered read, i.e. callingrgdx
with input uels, then the output uels will be essentially the same as the input uels. dim
It is a scalar value representing the dimension of the symbol.Apart from these necessary fields there are a few additional fields as well. They are as follows:
- field
If we are reading variables or equations, then it becomes useful to know which field we have read, i.e.,l/m/up/lo/s
. This information is displayed via this field in the form of a string. - ts
It display the explanatory text string associated with the symbol. This field only exists in the output structure if the'ts'
field is set as'true'
in the input structure. - te
It is an N dimensional cell array representing the text elements associated with each index of the set. This field only exists in the output structure if the'te'
field is set as true in the input structure and the symbol is a set.
See datalib example gdxmrw_rgdx01_init
.
wgdx
wgdx
is a MATLAB routine to create a GDX file containing data from MATLAB. Similar to the rgdx
routine, it takes a structure input and can write multiple symbols into a single GDX file with one call. The first argument is the file name of the GDX file to be created in string format; it can be with or without the '.gdx'
file extension. The rest of the arguments are structures, each containing data for different symbols to be written into the GDX file. The syntax for the call is:
>> wgdx('fileName', s1, s2 ...);
If the GDX file already exists in the MATLAB current directory, wgdx
will overwrite it; otherwise a new file will be created. After a successful run, it doesn't return anything back into MATLAB. Most of the fields of its input structures are the same as those of the rgdx
output structure. In the example below, we use the wgdx
routine to create foo.gdx
containing a set 'l'
and a parameter 'par'
.
>> s.name = 'l'; >> s.uels = {{'i1', 'i2', 'i3'}, {'j1', 'j2'}}; >> c.name = 'par'; >> c.type = 'parameter'; >> c.val = eye(3); >> c.form = 'full'; >> c.ts = '3 x 3 identity'; >> wgdx('foo', s, c)
The equivalent code in GAMS to create the above set 'l'
is:
Set
a / i1*i3 /
b / j1*j2 /
l(a,b);
l(a,b) = yes;
In the next subsection we will explain the input fields in detail.
Input Structure
The necessary fields in an input structure to represent a symbol are given below:
- name
It is a string representing the name of the symbol. - val
It represents the value matrix of the parameter or set. It can be entered in either full or sparse format, whichever is convenient to the user; the corresponding format must be specified in the'form'
field. By default the value matrix is assumed to be in sparse format. - type
It is a string input to specify the type of the symbol. Thewgdx
routine can write a set or parameter into the GDX file. In the previous example, we didn't specify the type for structure's'
because by default it is assumed to be a set. - form
This is a string input representing the format in which the val matrix has been entered. By default it is assumed that the data is specified in sparse format. uels
Similar to thergdx
uels field, this represents the local unique element listing of the symbol in an 1 x N cell array form. Each column of this cell array can contain string or double or both. Again we can make use of a handy function to create a UEL listing as mentioned previously in the section aboutrgdx
:>> guel = @(s,v) strcat(s,strsplit(num2str(v)));
As an example, we create a set using this function and the
wgdx
routine:>> s.name = 'l'; >> s.uels = {guel('i',1:20),guel('j',1:40)}; >> wgdx('foo',s);
The above code will create
foo.gdx
that contains a 2 dimensional set'l'
with set elements for each of the specified uels. If a user would like to create a set with set elements only at specified indices, one could do so by entering the indices in the val field. e.g.>> s.val = [1, 1; 2, 1; 3, 6; 4, 7]; >> wgdx('foo',s);
Now the above would result in a set with only elements existing in the indices specified in the
val
field. If a user enters a structure with only two fields, name and uels, as in the example given in the introduction of this section (structure s), thenwgdx
will create a full set corresponding to the global uels.The optional fields are:
- dim
This field is useful when a user wants to write a zero dimensional or 1 dimensional data in full format. As every data matrix in MATLAB is at least 2D, it becomes necessary to indicate its dimension for writing purposes. - ts
This is the text string that goes with the symbol. If nothing is entered then'MATLAB data from GDXMRW'
will be written in the GDX file.
See datalib example gdxmrw_wgdx01_init
.
Extended use
In this section, we will discuss a few examples of the MATLAB and GAMS interface. We will give a simple example of a nonlinear optimization problem that would benefit from this capability and describe the steps that are needed in order to use our interface in this application.
- Special values
Following example shows how special values are handled by this interface. It can be seen thatrgdx
can retrieve all these values from GDX file and display them appropriately in MATLAB.>> s.name = 'special'; >> s.form = 'full'; >> s.compress = true; >> x = rgdx('sample', s) x = name: 'special' type: 'parameter' dim: 1 val: [4x1 double] form: 'full' uels: {{1x4 cell}} >> x.val ans = -Inf NaN 3.141592653589793 Inf
- Variables and Equations
In an optimization problem, we are not only interested in level value of variables and equations but also in their marginal values, lower and upper bounds. This interface gives its user ability to read any of these values into MATLAB. By defaultrgdx
andgams
routines will read the level value of equations and variables but this can be changed very easily by using'field'
in input structure. Ingams
call user can also specify this in'$set matout'
statement. e.g.$set matout "'matsol.gdx', x.m, dual.lo=dl ";
In this case the marginal value of variable'x'
will be read and lower bound of dual variable will be read and stored in'dl'
. - Text string and Text elements
GAMS allows its user to enter text string and explanatory text elements and all GDX file contain this information as well. Following example shows how to get these text elements in MATLAB.>> s1.name = 'el'; >> s1.te = true; >> s1.ts = true; >> s1.compress = true s1 = name: 'el' te: 1 ts: 1 compress: 1 >> z = rgdx('sample', s1) z = name: 'el' type: 'set' dim: 2 val: [3x2 double] form: 'sparse' uels: {{1x2 cell} {1x2 cell}} ts: 'This is 2D set with text elements' te: {2x2 cell} >> z.te ans = 'element1' 'element2' '2.j1' [] >> z.val ans = 1 1 1 2 2 1
- String elements
One piece of information that may be needed within MATLAB is the modelstat and solvestat values generated by GAMS for the solves that it performed. This is easy to generate, and an example is given indo_status.m
. This example is generated by taking the standard gamslib trnsport example, and adding the following lines to the end:
Note that the relevant status numbers are stored in GAMS into the parameter returnStat which is then written to matsol.gdx and read back into MATLAB using the$set matout "'matsol.gdx', returnStat, str "; Set stat / modelstat, solvestat / str / 'grunt', '%system.title%' /; Parameter returnStat(stat); returnStat('modelstat') = transport.modelstat; returnStat('solvestat') = transport.solvestat; execute_unload %matout%;
rgdx
call.>> gamso.output = 'std'; >> gamso.form = 'full'; >> gamso.compress = true; >> s = gams('trnsport') s = 1 1
See datalib example gdxmrw_ext01_init
.
- Advanced Use: Plotting
One of the key features of the GAMS/MATLAB interface is the ability to visualize optimization results obtained via GAMS within MATLAB.
Some simple examples are contained with the program distribution. For example, a simple two dimensional plot with four lines can be carried out as follows. First create the data in GAMS and export it to MATLAB using thegams
routine (see APPENDIX C - Calling GAMS model from MATLAB for a detailed description of this routine).
We make an assumption that the user will write the plotting routines in the MATLAB environment. To create the plot in MATLAB, the following sequence of MATLAB commands should be input (saved asdo_plot.m
).The data is created using the following gams code.gamso.output = 'std';gamso.compress = true;gamso.form = 'full';[a,xlabels,legendset,titlestr] = gams('simple');figure(1)% Plot out the four lines contained in a;% format using the third argumentplot(a,'+-');% only put labels on x axis at 5 year intervalsxtick = 1:5:length(xlabels{1});xlabels{1} = xlabels{1}(xtick);set(gca,'XTick',xtick);set(gca,'XTickLabel',xlabels{1});% Add title, labels to axestitle(titlestr{1});xlabel('Year -- time step annual');ylabel('Value');% Add a legend, letting MATLAB choose positioninglegend(legendset{1},0);% match axes to data, add grid lines to plotaxis tightgrid$title Examples for plotting routines via MATLAB $set matout "'matsol.gdx', a, t, j, sys "; Set sys / '%system.title%' / t / 1990*2030 / j / a, b, c, d /; Parameter a(t,j); a("1990",j) = 1; loop(t, a(t+1,j) = a(t,j)*(1 + 0.04*uniform(0.2,1.8));); Parameter year(*); year(t) = 1989 + ord(t); * Omit some data in the middle of the graph: a(t,j)$((year(t) > 1995)*(year(t) <= 2002)) = na; execute_unload %matout%;
The following figure is an example created using this utility (and the MATLAB command print -djpeg
simple).

See datalib example GDXMRWPlotting01
.
MATLAB supports extensive hard copy output or formats to transfer data to another application. For example, the clipboard can be used to transfer meta files in the PC environment, or encapsulated postscript files can be generated. The help print
command in MATLAB details the possibilities on the current computing platform.
Scaling of pictures is also most effectively carried out in the MATLAB environment. The following code is an example of rescaling printed out. Note that the output of this routine is saved as a jpeg file "rescale.jpg"
.
Other examples of uses of the utility outlined in this paper can be found in the "m"
files:
that are contained in the distribution.
Acknowledgements
The authors would like to thank Alexander Meeraus and Michael R. Bussieck of GAMS corporation for constructive comments on the design and improvement of this tool. Thanks to Rishabh Jain for work on an earlier version.
APPENDIX A - Configuring GDXMRW
Installation
This section describes the installation procedure for all types of machines. The following section describes the testing procedure for verifying a correct installation.
First of all, you need to install both MATLAB and GAMS on your machine. For brevity, we will assume that the GAMS system (installation) directory is (for Windows)
C:\GAMS
and for non-Windows systems:
/usr/local/gams
All of the utilities come as a part of the GAMS distribution, so to use them you have only to add the GAMS directory to the MATLAB path. One way to do this is from the MATLAB command prompt, as follows:
>> addpath 'C:\GAMS'; savepath;
OR this can be done by following these steps:
- Start MATLAB
- Click on 'File' tab.
- Now click on 'Set Path'
- Click on 'Add Folder'
- Select GAMS directory and click 'OK'.
- Save it and then close it.
Testing
The GAMS system comes with some tests that you should run to verify the correct configuration and operation of the GDXMRW utilities. In addition, these tests create a log file that can be useful when things don't work as expected. To run the tests, carry out the following steps.
- Create a directory to run the tests in, e.g.
% mkdir \tmp
- Extract the test models and supporting files from the GAMS test library into the test directory.
% cd \tmp % testlib gdxmrw03 % testlib gdxmrw04 % testlib gdxmrw05 % testlib gdxmrw06 % testlib gdxmrw07
- Execute the GAMS files
gdxmrw03
,gdxmrw04
, andgdxmrw06
. The filesgdxmrw03
andgdxmrw04
test that thergdx
andwgdx
routines are working properly, andgdxmrw06
testirgdx
andiwgdx
. In addition to calling MATLAB in batch mode, they verify that the data are read and written as expected and give a clear indication of success or failure. - The GAMS file
gdxmrw05
tests thegams
MATLAB routine. Like the other tests, it can be run in batch mode. You can also run it interactively by starting MATLAB, makingtmp
the current directory, and running the script'testinst.m'
.>> testinst
In addition to messages indicating success or failure, this test produces a log filetestinstlog.txt
that will be useful in troubleshooting a failed test. - The GAMS file
gdxmrw07
helps in basic testing if you find that GDXMRW is not functioning at all.To use this, open Matlab, maketmp
the current directory and run gdxmrwSupport. You can then look over the outputs and identify if and where any errors or failures are occurring. Re-run the commands that result in failures via cut-and-paste, and examine the tests and the output carefully: perhaps a solution will present itself to you. If the issue persists, send the screen output and the resulting log file gdxmrwSupport.txt (located in the created folder tmp_gdxmrw) to GAMS Support (support@gams.com).
APPENDIX B - Utility functions: gdxWhos and gdxInfo
In this section, we'll describe two utilities, gdxWhos
and gdxInfo
, provided with GDXMRW that can be called from the MATLAB command prompt. The gdxWhos
function is patterned loosely after the MATLAB function whos
used to query .mat
files. It provides information about symbols in a specified GDX file. The only input argument is the name of a GDX file, which can be with or without the '.gdx'
extension. With no output argument, gdxWhos
lists the symbols in the specified GDX file in the MATLAB command prompt. When used with an output argument, information about the symbols in the GDX file is returned as an array of structures. This meta-data can be used in a variety of ways and is especially useful when programming.
An example GDXWhos call without an output argument is shown below:
>> gdxWhos('idx1_.gdx'); Symbol info of GDX idx1_.gdx Index Type Dim NRecs Name 1 Parameter 0 1 a0 2 Parameter 1 3 a1(5) 3 Parameter 2 4 a2(2,2) 4 Parameter 3 6 a3(3,5,2) 5 Parameter 10 256 a10(3,5,2,2,2,2,2,2,2,2)
The above output indicates that there are five indexed parameters in the file 'idx1_.gdx'
. The way we can recognize an indexed parameter is by noticing numbered arguments of the parameters under the 'Name'
column. For non-indexed data, the arguments will appear either as the name of a set or by a '*'. For example, the first three symbols are non-indexed and the last symbol is indexed in the output from this call:
>> gdxWhos('fake.gdx'); Symbol info of GDX fake.gdx Index Type Dim NRecs Name 1 Set 1 3 i(*) 2 Parameter 1 3 a(i) 3 Set 1 3 d_i_m__3(*) 4 Parameter 1 3 aa(3)
The GDXInfo function is patterned after the gdxdump
utility in GAMS. It lists and dumps all of the data values for each symbol in the specified GDX file to the MATLAB console. The only input argument is the name of a GDX file, which can be with or without the '.gdx'
extension. It accepts no output arguments. It is especially useful to see the entire contents of smaller GDX files. To use the function, we could simply enter a command such as gdxInfo('idx1_')
in the MATLAB command prompt, and the entire data contents would be displayed in the console. Further information about what is displayed can be found in the GAMS documentation of the gdxdump
utility.
APPENDIX C - Calling GAMS model from MATLAB
Until now we have discussed the data Import/Export utility between MATLAB and GAMS. In this section, we will discuss a new MATLAB utility 'gams'
that initializes a GAMS model with MATLAB data then executes GAMS on that model and bring the results back into MATLAB. This 'gams'
routine is based on the same design as rgdx
and wgdx
but instead it does everything in one call. This routine can take multiple input arguments and can return multiple output arguments. Its standard syntax is as follows:
>> [x1, x2, x3] = gams('model', s1, s2.., c1, c2..);
Here note that the first argument of gams
is the GAMS model name plus any user specific command line settings. If a user wants to solve the given model (in this case found in qp.gms
) using a different solver then it can be done by adding that solver to the GAMS model name as "qp nlp=baron"
. This feature allows a user to change the execution time behavior of the model.
The rest of the input arguments of GAMS are structures. Their positioning is not important. These structures are of two kinds; one is similar to the input structure of wgdx
and the other structure has just two string fields, name and val. This latter structure is used to set or overwrite values in the model using the "$set"
variables syntax of GAMS. We will explain it in detail a later section.
The first step is to generate a working GAMS model. For example, we can set up a simple model file to solve a quadratic program that minimizes ½ xTQx+cTx subject to Ax≥b and x≥0.
The GAMS model for this quadratic problem is as given in the following code.
$set matout "'matsol.gdx', x, dual, obj, returnStat ";
Set
i / 1*2 /
j / 1*3 /;
Alias (j1,j);
Parameter
Q(j,j1) / 1 .1 1.0
2 .2 1.0
3 .3 1.0 /
A(i,j) / 1 .1 1.0
1 .2 1.0
1 .3 1.0
2 .1 -1.0
2 .3 1.0 /
b(i) / 1 1.0
2 1.0 /
c(j) / 1 2.0 /;
Variable obj;
Positive Variable x(j);
Equation cost, dual(i);
cost.. obj =e= 0.5*sum(j, x(j)*sum(j1, Q(j,j1)*x(j1))) + sum(j, c(j)*x(j));
dual(i).. sum(j, A(i,j)*x(j)) =g= b(i);
Model qp / cost, dual /;
$if exist matdata.gms $include matdata.gms
solve qp using nlp minimizing obj;
Set stat / modelStat, solveStat /;
Parameter returnStat(stat);
returnStat('modelStat') = qp.modelstat;
returnStat('solveStat') = qp.solvestat;
execute_unload %matout%;
This GAMS qp model can be executed directly at the command prompt using the following command
gams qp (for Unix/Linux)
or
gams.exe qp (for Windows)
or the user can simply hit the run button in the GAMSIDE. The optimal value is 0.5. In order to run the same model within MATLAB and return the solution vector x back into the MATLAB workspace, no change is required to the GAMS file. In MATLAB, all you have to do is to execute the following command:
>> x = gams('qp');
This command will first collect the input structure data and create 'matdata.gdx'
and 'matdata.gms'
that contains include statements for the symbols written in a file matdata.gdx
. In the previous example there is no structural input, so an empty 'matdata.gdx'
file will be created and 'matdata.gms'
will have just have a load statement for the GDX file but no load statements for any symbol. This is done to prevent any undesirable loading of data in the main model if there had already existed a 'matdata.gdx'
or 'matdata.gms file'
. After creating these two files then the 'gams'
routine will execute "gams qp"
using a system call. When this model is executed, another file 'matsol.gdx'
will be created because of the execute_unload
statement in the last line of the model. Here it should be noted that any model that you want to execute using the MATLAB gams
routine should contain something like
$set matout "'matsol.gdx', x, dual, obj, returnStat ";
either as the first line, or somewhere near the start of the model file. This is a standard gams
$set
statement, setting the value of the local variable 'matout'
. The reason to have this statement near the start of the gms file is that the gams
routine searches the file from the beginning for "$set matout"
in the gms file. As these files can be very large, it is wise to have this statement near the start of the file. In this statement 'fileName'
is the gdx file name that will be created containing symbols 'x1'
, 'x2'
, etc. These symbols can then be exported to MATLAB. The last line of the model should always be
execute_unload %matout%;
The purpose of setting the first and last line of the model in this manner is to specify what data the user wants to export to MATLAB in a "header"
of the model. As MATLAB does not give any information about the output arguments except the number of expected arguments, we have to specify what data to export to MATLAB in the GAMS model with minimum modification to the existing model. In the previous example, there is only one output argument, thus the gams
routine will get data for its first element from the output gdx file and store it in the MATLAB output argument.
If there is more than one output argument, e.g.,
>> [x, u] = gams('qp');
then the gams
routine will read the output gdx file and store its first element information of the GDX file as the first output argument of MATLAB, i.e. 'x'
, and the second element information of the GDX file in the second output argument of MATLAB, i.e. 'u'
and so on. If the number of MATLAB output arguments is greater than the number of elements in the GDX file then gams
will throw an error.
See datalib example gdxmrw_qp4
.
Input Structure
As mentioned earlier, the gams
routine takes input arguments in structured form. It allows two different types of structure input. One contains the symbol data similar to the wgdx input structure, to be exported to the GDX file. The other structure will just have two string fields 'name' and 'value'. Example use:
>> s.name = 'Q'; >> s.val = eye(3); >> s.form = 'full'; >> m = struct('name','m','val','2'); >> [x] = gams('qpmcp',s, m);
In this example both 's'
and 'm'
are structures but 'm'
has only two fields and both are strings. The gams
routine will use the 's' structure to create a 'matdata.gdx'
file and 'm'
to modify the execution command line to include "--m=2" at the end i.e. a command that when executed will be "gams qpmcp --m=2"
The structure 's'
is the same as the input structure for wgdx
but with two important differences. Firstly, it can be seen in the above example that 's'
doesn't have a 'type'
field. In wgdx
we assume the type to be 'set' by default, but in the gams
routine the type is assumed to be 'parameter'
by default. The second change is an optional additional input field (in addition to those given in Section Input Structure) called "load"
.
- load
It is a string input representing how the corresponding data will be loaded into the GAMS program. Depending on the value of the global option"gamso.input"
(see next section) the input data will be read into GAMS in different ways. Suppose the input structure's'
has a'name'
field called'foo'
. By default (wheregamso.input = 'compile'
), the filematdata.gms
will
The GAMS parameter (or set) foo will be replaced by the data that is in the$loadR foo
'matdata.gdx'
container called'foo'
. If the data has been initialized before in the model, this will replace that initial data with the new data from ‘matdata.gdx’. The option can also be explicitly set using
There are two other compile time load options, namelys.load = `replace'
'initialize'
and'merge'
. The first is only valid if the parameter values have not been initialized in the GAMS file, otherwise an error is thrown. It uses the GAMS syntax
The merge option is valid when the GAMS file being run has already initialized the parameter values. The new values in the MATLAB structure$load foo
's'
are merged into the parameter simply overwriting existing values with the new values given. Explicitly, the'matdata.gms'
file contains the statement
to direct GAMS accordingly.$loadM foo
Finally, if gamso.input = ‘exec’, the loading will occur at execution time. In this case,s.load = 'initialize'
is not a valid input, the default setting iss.load = 'replace'
which carries out
and the alternative settingexecute_load "matdata.gdx" foo
s.load = 'merge'
carries out
In this way, the data is loaded at execution time and performs an appropriate replace or merge.execute_loadpoint "matdata.gdx" foo
Global input to change default behavior
Until now we have seen how to specify different input to the gams
routine and in this section we will see how to change the default behavior of a gams
call. This can be done by creating a structure 'gamso'
in the current workspace and adding different fields to that structure. There are currently nine fields that can be set in that structure that affect the behavior of the program. Except the uels field, all other string fields take case insensitive data. These are as follows:
gamso.output
By default, output of thegams
routine will be in structure form but it might be the case that a user is only interested in the data matrix, i.e., the val field of that structure. This can be done by settinggamso.output
as'std'
. This will give only the value matrix as output. If this is not set to'std'
then output will be in the structure form described in thewgdx
section.>> gamso.output = 'Std'; >> x = gams('qp nlp=baron') x = 0.5000
gamso.input
By default, the interface updates data at compile time. Thus, if execution time updates are made to the parameters before the line'$include matdata.gms'
these may override the data that is provided in'matdata.gms'
(i.e. from the command line). This may not be desirable. If you wish to perform execution time updates to the data, you should setgamso.input
to'exec'
.gamso.write_data
If this is set to'no'
, then all parameters on the call togams
are ignored, except the program name. This is useful for dealing with large datasets. Consider the following invocation:
The first call generates a filex = gams('largedata','A'); y = gams('resolve','A');
'matdata.gms'
containing the elements of the matrixA
for use in thelargedata.gms
program. The second call rewrites a new'matdata.gms'
file that again containsA
. If we wish to save writing outA
the second time we can use the following invocation:
or the equivalent invocation:x = gams('largedata','A'); gamso.write_data = 'no'; y = gams('resolve','A'); clear gamso;
x = gams('largedata','A'); gamso.write_data = 'no'; y = gams('resolve'); clear gamso;
gamso.show
This is only relevant on a Windows platform. This controls how the'command box'
that runs GAMS appears on the desktop. The three possible values are:'minimized'
(default): The command prompt appears iconified on the taskbar.'invisible'
: No command prompt is seen.'normal'
: The command prompt appears on the desktop and focus is shifted to this box.
gamso.path
This option is used to specify fully qualified path for the gams executable. This is very useful if you have multiple versions of GAMS installed on your system and want to make sure which version you are running for thegams
call. e.g.>> gamso.path = 'C:\Program Files\GAMS23.4GAMS.exe';
The output of gams
is similar to rgdx
but unlike the rgdx
gams
routine it doesn't take input specific to a particular symbol. Thus it becomes important to implement a way to change the default behavior of the output. This can be achieved by adding following field to the global structure 'gamso'
. All these fields behave similar to that described in rgdx
and take the same input as of rgdx
.
gamso.compress
gamso.form
gamso.uels
gamso.field
This is a global option however.