Usage

gdxmerge filepattern1 filepattern2 .... filepatternN {options}

Each file pattern represents a file name or a wildcard representation using ? and *. A parameter of the form @filename will process the commands from the text file specified. The result of the GDXMERGE execution will be written to a file called merged.gdx, unless this default is overwritten by the output option.

The .gdx file extension can be omitted. Files without a full path name are assumed to be in the current directory when using a command prompt. When using the GAMS IDE, these files are assumed to be in the current project directory.

Options

The following options can be specified:

id = <ident1>, <ident2>... (default = none)

Only merge the symbols ident1, ident2, ...

exclude = <ident1>, <ident2>... (default = none)

Merge all symbols except for ident1, ident2`, ...

big = <integer>

The size for big symbols.

strict = true|false (default = false)

The program terminates with an error (non-zero return code) if input files specified cannot be found or a file pattern results in no files. An already existing output file also triggers an error if strict is set to true.

output = fileName (default = merged.gdx)

The optional output file name.

All symbols with matching type and dimension will be merged. By specifying the parameter id=ident1 the merge process will only be performed for the identifier(s) specified, while exclude=ident1 indicates that all symbols should be merged except for the ones specified in the exclude list. Note that the two options id and exclude are mutually exclusive.

By default, the program reads all GDX once and stores all data in memory before writing the merged.gdx file. The big parameter is used to specify a cutoff for symbols that will be written one at a time. Each symbol that exceeds the size will be processed by reading each GDX file and only process the data for that symbol. This can lead to reading the same GDX file many times, but it allows the merging of large data sets. The formula used to calculate the cutoff is:

dimension * totalNumberOfElements.

The calculated value is doubled for variables and equations.

In addition to the symbols written, a set is added to the GDX file representing all the files processed during the merge operation. The name of the set is Merged_set_1, and is made unique by changing the number. The explanatory text for each set element contains the date and time of the GDX file processed.

Note

The file merged.gdx, or the file specified with the output parameter, will never be used in a merge operation even if the name matches a file pattern.
Symbols with dimension 20 cannot be merged, because the resulting symbol will have dimension 21 which exceeds the maximum dimension allowed by GAMS.

Examples

Merging several GDX Files

In this example, we solve the [trnsport] model from the GAMS Model Library using different LP solvers. After each run, we write all symbols to a GDX file and merge the files into one file. The variable X is read from the merged file and displayed.

$call gamslib trnsport
$call gams trnsport lp=cplex gdx=cplex
$call gams trnsport lp=xpress gdx=xpress
$call gams trnsport lp=conopt gdx=conopt
$call gams trnsport lp=minos gdx=minos
$call gams trnsport lp=snopt gdx=snopt
$call gdxmerge *.gdx

Variable AllX(*,*,*);
$gdxIn merged.gdx
$load AllX=X
$gdxIn

option  AllX:5:1:2;
display AllX.L;

The display statement generates the following output in the listing file:

----     22 VARIABLE AllX.L  shipment quantities in cases

           seattle     seattle   san-diego   san-diego
          new-york     chicago    new-york      topeka

conopt               300.00000   325.00000   275.00000
cplex     50.00000   300.00000   275.00000   275.00000
minos     50.00000   300.00000   275.00000   275.00000
snopt     50.00000   300.00000   275.00000   275.00000
xpress               300.00000   325.00000   275.00000

Note that the different solutions are combined into a single symbol of a higher dimension. The filenames (resp. the solver used) are added as unique elements.
Instead of using the display statement, we can also use the GAMS IDE or GAMS Studio to display the merged.gdx file. The following figure shows the contents of merged.gdx after selecting the level subfield of the variable to be displayed and arranging the display:

Output file merge.gdx displayed in GAMS Studio

This example is also part of the GAMS Data Utilities Library, see model [GDXMERGEExample17] for reference.

Recursively Merging GDX Files

Some users generate data scenarios for a model in a hierarchical way and store the results in a directory tree:

Scenario directory tree

The following example demonstrates how to use three independent scenario scalars a, b, and c with values low, medium, and high to generate 3*3*3=27 scenarios, store these in a directory tree, and merge them recursively. The resulting GDX file has for each scenario scalar a new dimension which makes this data especially suitable for viewing in a pivot table:

----     39 PARAMETER p  commodity price

                               new-york     chicago      topeka

a_high.b_high.c_high.report        0.25        0.17        0.14
a_high.b_high.c_low .report        0.20        0.14        0.11
a_high.b_high.c_med .report        0.23        0.15        0.13
a_high.b_low .c_high.report        0.25        0.17        0.14
a_high.b_low .c_low .report        0.20        0.14        0.11
a_high.b_low .c_med .report        0.23        0.15        0.13
a_high.b_med .c_high.report        0.25        0.17        0.14
a_high.b_med .c_low .report        0.20        0.14        0.11
a_high.b_med .c_med .report        0.23        0.15        0.13
a_low .b_high.c_high.report        0.25        0.17        0.14
a_low .b_high.c_low .report        0.20        0.14        0.11
a_low .b_high.c_med .report        0.23        0.15        0.13
...

The GAMS program that creates the directory structure and runs the model for all 27 scenarios looks as follows:

set a,b,c;
parameter
    ascen(a<) / a_low 0.9, a_med 1.0, a_high 1.1 /
    bscen(b<) / b_low 0.9, b_med 1.0, b_high 1.1 /
    cscen(c<) / c_low 0.9, c_med 1.0, c_high 1.1 /;
    
loop((a,b,c),
   put_utility 'shell' / 'mkdir ' a.tl:0 '\' b.tl:0 '\' c.tl:0
   put_utility 'exec'  / 'gams t.gms lo=2 -idir "%gams.wdir%" -curdir ' a.tl:0 '/' b.tl:0 '/' c.tl:0 ' --a ' ascen(a):3:1 ' --b ' bscen(b):3:1 ' --c ' cscen(c):3:1
);

The program creates the directory tree (by creating the leave nodes of the tree), changes into the leave directory (curDir) and calls GAMS (using iDir to point to the starting location). The program t.gms is almost an identical copy of the [trnsport] model with the last few lines changed:

* ...
a(i) = %a%*a(i);
b(j) = %b%*b(j);
c(i,j) = %c%*c(i,j);

solve transport using lp minimizing z;

scalar tmodelstat; 
tmodelstat = transport.modelstat;
parameter p(j) 'commodity price'; 
p(j) = demand.m(j);
execute 'test -d results || mkdir results';
execute_unload 'results/input', a, b, c;
execute_unload 'results/report', x, z, p, tmodelstat;

This code saves the results of a run in to GDX files input.gdx and report.gdx in a results subdirectory. After the program has executed the 27 runs of t.gms it executes some Python code that walks recursively through the directory tree and merges GDX files it finds in a results folder in the leaf nodes percolates the results up in the tree by calling gdxmerge on each high node level. The Python function os.walk with argument topdown=False ensure the walking of the tree in the correct order. The complete example can be found in the GAMS Model Library in model [scenmerge]:

embeddedCode Python:
import os
gd = -1
for root, dirs, files in os.walk(os.path.normpath(r"C:\Users\test\allscen\ ".rstrip()), topdown=False):
  if 'results' in dirs:
    if gd == -1:
      gd = root.count(os.path.sep) 
      gdr = root;
    elif not gd == root.count(os.path.sep):
      raise NameError('results subdirs found at different depths in the tree: ' + root + ' and ' + gdr)
    cmd = 'cd "' + root + '" && gdxmerge results' + os.path.sep + '* output=' + root.split(os.path.sep)[-1]
    print('Merging results in ' + root)
    if not 0 == os.system(cmd + '>' + os.devnull):
      raise NameError('problems running: ' + cmd)
  elif root.count(os.path.sep) < gd:
    input = ''
    for d in dirs:  
      if os.path.isfile(os.path.join(root, d, d + '.gdx')):
        input = input + ' ' + os.path.join(d, d + '.gdx')
    if len(input):
      cmd = 'cd "' + root + '" && gdxmerge' + input + ' output=' + root.split(os.path.sep)[-1]
      print('Merging' + input + ' in ' + root)
      print(cmd)
      if not 0 == os.system(cmd + '>' + os.devnull):
        raise NameError('problems running: ' + cmd)
print('All done. Final result file: ' + root.split(os.path.sep)[-1] + '.gdx')
endEmbeddedCode

Table of Contents

Usage

Options

Examples

Merging several GDX Files

Recursively Merging GDX Files