Embedded Code Facility
Note
This feature is currently in beta status.

Motivation

GAMS uses relational data tables as a basic data structure. With these, GAMS code for parallel assignment and equation definition is compact, elegant, and efficient. However, traditional data structures (arrays, lists, dictionaries, trees, graphs, ...) are not natively or easily available in GAMS. Though it is possible to represent such data structures in GAMS, the GAMS code working with such structures can easily become unwieldy, obfuscating, or inefficient. Also, in the past it was not easy to connect libraries from other systems for special algorithms (e.g. graph algorithms, matrix operations, ...) to GAMS without some data programming knowledge and a deep understanding of internal representation of GAMS data (e.g. the GDX API).

The Embedded Code Facility addresses this need and extends the connectivity of GAMS to other programming languages. It allows the use of external code (e.g. Python) during compile and execution time. GAMS symbols are shared with the external code, so no communication via disk is necessary. It utilizes an API (this API will be published in one of the next releases so users can extend the embedded code to other languages/systems) and additional source code for common tasks so that the user can concentrate on the task at hand and not the mechanics of moving data in and out of GAMS.

Concept

As pointed out in section Motivation, the main idea of the Embedded Code Facility is to enable the use of external code in GAMS and give this code direct in-memory access to the GAMS database (or better: to GAMS symbols, namely sets, parameters, variables, and equations). This can be done by defining sections in the GAMS code which contain no GAMS code, but code written in a defined external code. These sections can be used at both GAMS compile and execution time (compare section GAMS Compile Time and Execution Time Phase). Details about how to do this can be found in the Syntax section.

Note
It is planned to extend this feature to different programming languages. However, at the moment only Python is supported.

Also, the system provides some help to develop and debug the external code independent of GAMS first. More about this topic can be found in section Troubleshooting Embedded Python Code.

The communication of the data between the GAMS part of the program and the embedded code part was inspired by the existing interface to GDX in many ways. So the system allows to access specific symbol records by both labels and label indices. Also, it is possible to decide if data changed in the embedded code should replace the GAMS data or get merged with it and just like loading data from GDX, one can decide if data from embedded code should change the GAMS database filtered or domain checked. How to do these things is explained in detail in section Python.

Simple Example

In a very first example we look into some Python code that helps to split some label names that are already present in GAMS. We do this at compile time in order to read the broken up pieces as individual sets (country and city) into GAMS plus some mapping sets (mccCountry and mccCity) between the original labels and the new labels. The compile-time embedded code starts with a $onEmbeddedCode followed by the type of code to expect (Python:). Python is currently the only code that works. GAMS plans to add other (even compiled) languages in the future. The lines between $onEmbeddedCode and $offEmbeddedCode is Python code. We do not want to go into Python details, but the first few lines initialize some empty Python list objects (mccCountry and mccCity) as well as some empty Python set objects. In the Python for loop that follows we iterate over all individual labels of the GAMS set cc. Python gets access to the GAMS set cc via the member function get of the implicitly defined Python object gams. get returns an object that is iterable and can be used in a Python for loop. The type of the records one gets depend on the dimensionality and type of the GAMS symbol. In the loop body we use the Python split function to extract the first (r[0] is country) and second (r[1] is city) part of the label. The three strings cc, r[0], r[1] are used to build up the Python list objects that store the information for the maps mccCountry and mccCity and the Python set objects that store the labels for the new sets country and city. The Python set has the advantage to store a label just once even if we add it multiple times. Python prepares to send items back to GAMS via the gams member function set that can deal with both Python list and Python set objects. The command $offEmbeddedCode is followed by a list (without separating commas) of GAMS symbols that instructs the GAMS compiler to read these symbols back.

Note that GAMS syntax is case insensitive while Python is case sensitive. Hence, the strings that represent the GAMS symbol names in the Python code can have any casing (e.g. gams.get("cc") or gams.get("CC")), while the corresponding Python objects need to have consistent casing throughout the Python code.

The following code presents the entire embeddedSplit example from the GAMS Data Utilities Library:

Set cc      / "France - Paris", "France - Lille", "France - Toulouse"
              "Spain - Madrid", "Spain - Cordoba", "Spain - Seville", "Spain - Bilbao"
              "USA - Washington DC", "USA - Houston", "USA - New York",
              "Germany - Berlin", "Germany - Munich", "Germany - Bonn" /
    country / system.empty /
    city    / system.empty /
    mccCountry(cc,country), mccCity(cc,city);

$onEmbeddedCode Python:
  mccCountry = []
  mccCity = []
  country = set()
  city = set()
  for cc in gams.get("cc"):
    r = str.split(cc, " - ", 1)
    mccCountry.append((cc,r[0]))
    mccCity.append((cc,r[1]))
    country.add(r[0])
    city.add(r[1])
  gams.set("country",country)
  gams.set("city",city)
  gams.set("mccCountry",mccCountry)
  gams.set("mccCity",mccCity)
$offEmbeddedCode country city mccCountry mccCity

Option mccCountry:0:0:1, mccCity:0:0:1;
Display country, city, mccCountry ,mccCity;

The display in the listing file looks as follows:

----     25 SET country  
Spain  ,    USA    ,    Germany,    France 

----     25 SET city  
Berlin       ,    Bilbao       ,    Cordoba      ,    Madrid       
New York     ,    Washington DC,    Paris        ,    Houston      
Munich       ,    Lille        ,    Seville      ,    Bonn         
Toulouse     

----     25 SET mccCountry  
France - Paris     .France 
France - Lille     .France 
France - Toulouse  .France 
Spain - Madrid     .Spain  
Spain - Cordoba    .Spain  
Spain - Seville    .Spain  
Spain - Bilbao     .Spain  
USA - Washington DC.USA    
USA - Houston      .USA    
USA - New York     .USA    
Germany - Berlin   .Germany
Germany - Munich   .Germany
Germany - Bonn     .Germany

----     25 SET mccCity  
France - Paris     .Paris        
France - Lille     .Lille        
France - Toulouse  .Toulouse     
Spain - Madrid     .Madrid       
Spain - Cordoba    .Cordoba      
Spain - Seville    .Seville      
Spain - Bilbao     .Bilbao       
USA - Washington DC.Washington DC
USA - Houston      .Houston      
USA - New York     .New York     
Germany - Berlin   .Berlin       
Germany - Munich   .Munich       
Germany - Bonn     .Bonn         

The second example demonstrates the use of embedded code at execution time. The syntax for the execution time embedded code in this example is identical to the compile time variant with the exception of the keywords that start and end the embedded code section: embeddedCode and endEmbeddedCode. It is important to understand that the execution of the code happens at GAMS execution time, so e.g. no new labels can be produced and send back to GAMS. In this example we use some Python code to generate a random permutation of set elements of set i and store this in a two dimensional set p. In this example we do not use a loop to iterate through the elements of a GAMS system but make use of the fact that the Python object returned by gams.get("i") is iterable and can in its entirety be stored in the Python list with name i with the short and powerful command i = list(gams.get("i")). The permutation of elements in list p which is a copy of list i is created by the Python statement random.shuffle(p). The following code presents the entire example:

Set i /i1*i10/
    p(i,i) "permutation";

embeddedCode Python:
  import random
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
  gams.set("p", p)
endEmbeddedCode p

option p:0:0:1; 
display p;

The display in the listing file looks as follows:

----     11 SET p  permutation
i1 .i1 
i2 .i7 
i3 .i5 
i4 .i2 
i5 .i10
i6 .i6 
i7 .i9 
i8 .i4 
i9 .i8 
i10.i3 

Syntax

This section explains the GAMS functions/keywords which were introduced to enable the Embedded Code Facility. The first subsection deals with the syntax for compile time, the second with the syntax for execution time (compare section GAMS Compile Time and Execution Time Phase).

Compile Time

There are three dollar control options to start an embedded code section for Python at compile time:

$onEmbeddedCode  Python: [arguments]
$onEmbeddedCodeS Python: [arguments]
$onEmbeddedCodeV Python: [arguments]

Lines following one of the above statements are passed on to a Python interpreter until this dollar control option, which ends the embedded code section at compile time, is hit:

$offEmbeddedCode {output symbols}

These dollar control options are explained here in more detail. An example which uses them can be seen above.

Note
  • The optional arguments from the $onEmbeddedCode[S|V] statement can be accessed as gams.arguments in the Python code.
  • The optional output symbols from the $offEmbeddedCode statement need to be set using the function gams.set in the Python code.
  • More about the specific GAMS Python syntax can be found below.

Execution Time

At execution time an embedded code section is started with one of these statements:

embeddedCode  Python: [arguments]
embeddedCodeS Python: [arguments]
embeddedCodeV Python: [arguments]

Similar to the compile time alternatives $onEmbeddedCode[S|V], the first two variants are synonyms which allow parameter substitution in the Python code that follows, while the last variant does not allow this but passes the code verbatim to the Python interpreter. The optional arguments in all three variants can be accessed in the Python code that follows as gams.arguments (see section Python for more details). Lines following one of the above statements are passed on to the Python interpreter until one of the following two statements, which end the embedded code section at execution time, is hit:

endEmbeddedCode   {output symbols}
pauseEmbeddedCode {output symbols}

Both statements end the embedded code section and switch back to GAMS syntax in the following lines. Also, both statements can be followed by a GAMS symbol or a list of GAMS symbols which would get updated in the GAMS database after the Python code got executed. If output symbols are specified, they need to be set in the embedded code before using the function gams.set (see section Python for more details). Now to the difference between the two statements: As the names suggest endEmbeddedCode ends the embedded code section, while pauseEmbeddedCode pauses it. If it ends, it cannot be continued at a later point, because some internal resources get freed, it could just be re-initiated (e.g. by another embeddedCode statement). If the embedded code section got paused, it could also be continued at a later stage in the GAMS program, which means that no reinitialization is needed and Python symbols which were defined before the pause are still available when continuing. Note that continuing a paused embedded code section is not possible after a solve statement if SolveLink is set to 0. Since GAMS does not stay in memory in case of SolveLink=0 the state of the embedded code environment is lost. By using a different value (e.g. SolveLink=2), GAMS stays in memory and a paused embedded code environment can be continued after the solve statement has carried out.

To continue a previously paused embedded code section one of the following statements is used:

continueEmbeddedCode  [handle]: [arguments]
continueEmbeddedCodeS [handle]: [arguments]
continueEmbeddedCodeV [handle]: [arguments]

As seen before, the first two variants are synonyms which allow parameter substitution in the embedded code that follows, while the last variant does not allow this but passes the code verbatim to the interpreter. Also as seen above when discussing EmbeddedCode[S|V], the optional arguments in all three variants can be accessed in the embedded code that follows as gams.arguments (see section Python for more details). New in these statements is the optional handle. If omitted, the last code section that was paused will be continued. However, sometimes one might need to maintain different embedded code sections active in parallel and independent of each other. In order to use this facility, it is required to set PyMultInst to 1. Note that this setting might cause problems when using third party modules and packages (e.g. numpy or modules that make use of it) and might also impact the performance. There is a new function to store a handle of the last embedded code section that was executed which could then later be used to continue a specific paused code section:

handle = embeddedHandle;

An example of how this can be used can be seen in the GAMS Datalib model [embeddedMultiInstance]. A simplified use of just embeddedCode and endEmbeddedCode can also be seen in a simple example above.

Note
The combination of pauseEmbeddedCode and continueEmbeddedCode[S|V] can be particularly useful, when there is an expensive initialization (e.g. to import other code parts) which can be done only once but further execution needs to be done many times (e.g. inside a loop).

Python

The Python class ECGamsDatabase is the interface between GAMS and Python. An instance of this class is automatically created when an embedded code section is entered and can be accessed using the identifier gams. The following methods can be used in order to interact with GAMS:

gams.get(symbolName, keyType=KeyType.STRING, keyFormat=KeyFormat.AUTO, valueFormat=ValueFormat.AUTO, recordFormat=RecordFormat.AUTO)

This method retrieves an iterable object representing the symbol identified with symbolName. Typically there are two possibilities to access the records. Iterating using e.g. a for loop provides access to the individual records. By calling list() on the iterable object, a list containing all the data is created. Several optional parameters can be used in order to modify the format of the retrieved data:

  • keyType: Determines the data type of the keys. It can be either KeyType.STRING (labels) or KeyType.INT (label indexes). The default setting is KeyType.STRING.
  • keyFormat: Specifies the representation of the keys. Possible values are as follows:
    • KeyFormat.TUPLE: Encapsulate keys in a tuple
    • KeyFormat.FLAT: No encapsulation
    • KeyFormat.SKIP: Keys are skipped and do not appear in the retrieved data
    • KeyFormat.AUTO (default): Depending on the dimension of the GAMS symbol, a default format is applied:
      • Zero dimensional/scalar: KeyFormat.SKIP
        • One dimensional: KeyFormat.FLAT
        • Multi dimensional: KeyFormat.TUPLE
  • valueFormat: Specifies the representation of the values. Possible values are as follows:
    • ValueFormat.TUPLE: Encapsulate values in a tuple
    • ValueFormat.FLAT: No encapsulation
    • ValueFormat.SKIP: Values are skipped and do not appear in the retrieved data
    • ValueFormat.AUTO (default): Depending on the type of the GAMS symbol, a default formats is applied:
      • Set: ValueFormat.SKIP
        • Parameter: ValueFormat.FLAT
        • Variable/Equation: ValueFormat.TUPLE
  • recordFormat Specifies the encapsulation of records into tuples. Possible values are as follows:
    • RecordFormat.TUPLE: Encapsulates every record in a tuple
    • RecordFormat.FLAT: No encapsulation. Throws an exception if it can not be applied. It is guaranteed that the length of a retrieved Python list is equal to the number of records of the corresponding GAMS symbol. This principle leads to an incompatibility of RecordFormat.FLAT whenever a record consists of more than one item (e.g. multi dimensional symbols, variables and equations which have five numeric values).
    • RecordFormat.AUTO (default): Depending on the number of items that represent a record, a default format is applied. If possible this is always RecordFormat.FLAT.

GAMS special values NA, INF, and -INF will be mapped to IEEE special values float('nan'), float('inf'), and float('-inf'). GAMS special value EPS will be either mapped to 0 or to the small numeric value 4.94066E-300 depending on the setting of flag gams.epsAsZero.

The following Python code shows some examples of gams.get and illustrates the use of different formats:

Set i / i1 text 1, i2 text 2 /
    j / j1*j2 /

Scalar p0 /3.14/;
Parameter p1(i)   / #i 3.14 /
          p2(i,j) / i1.#j 3.14 /
Variable v0       / fx 3.14 /;
equation e1(i)    / #i.fx 3.14 /
         e2(i,j)  / i1.#j.fx 3.14 /;

$onEmbeddedCode Python:
  # scalar parameter
  l = list(gams.get('p0'))
  assert l == [3.14], "error"
  l = list(gams.get('p0', recordFormat=RecordFormat.TUPLE))
  assert l == [(3.14,)], "error"

  # one dimensional parameters:
  l = list(gams.get('p1'))
  assert l == [("i1", 3.14), ("i2", 3.14)], "error"
  l = list(gams.get('p1', keyFormat=KeyFormat.TUPLE))
  assert l == [(("i1",), 3.14), (("i2",), 3.14)], "error"
  l = list(gams.get('p1', valueFormat=ValueFormat.TUPLE))
  assert l == [("i1", (3.14,)), ("i2", (3.14,))], "error"
  l = list(gams.get('p1', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE))
  assert l == [(("i1",), (3.14,)), (("i2",), (3.14,))], "error"

  # two dimensional parameter:
  l = list(gams.get('p2'))
  assert l == [(('i1', 'j1'), 3.14), (('i1', 'j2'), 3.14)], "error"
  l = list(gams.get('p2', keyFormat=KeyFormat.FLAT))
  assert l == [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)], "error"

  # one dimensional sets:
  l = list(gams.get('i'))
  assert l == ['i1', 'i2'], "error"
  l = list(gams.get('i',valueFormat=ValueFormat.FLAT))
  assert l == [('i1', "text 1"), ('i2', "text 2")], "error"

  # scalar variables/equations
  l = list(gams.get('v0'))
  assert l == [(3.14, 0, 3.14, 3.14, 1)], "error"

  # one dimensional variables/equations:
  l = list(gams.get('e1'))
  assert l == [("i1", (3.14, 0, 3.14, 3.14, 1)), ("i2", (3.14, 0, 3.14, 3.14, 1))], "error"
  l = list(gams.get('e1', valueFormat=ValueFormat.FLAT))
  assert l == [("i1", 3.14, 0, 3.14, 3.14, 1), ("i2", 3.14, 0, 3.14, 3.14, 1)], "error"
  l = list(gams.get('e1', keyFormat=KeyFormat.TUPLE))
  assert l == [(("i1",), (3.14, 0, 3.14, 3.14, 1)), (("i2",), (3.14, 0, 3.14, 3.14, 1))], "error"

  # two dimensional variables/equations:
  l = list(gams.get('e2'))
  assert l == [(("i1", "j1"), (3.14, 0, 3.14, 3.14, 1)), (("i1", "j2"), (3.14, 0, 3.14, 3.14, 1))], "error"
  l = list(gams.get('e2', keyFormat=KeyFormat.FLAT, valueFormat=ValueFormat.FLAT))
  assert l == [("i1", "j1", 3.14, 0, 3.14, 3.14, 1), ("i1", "j2", 3.14, 0, 3.14, 3.14, 1)], "error"

  # using label indexes instead of labels
  l = list(gams.get('p1', keyType=KeyType.INT))
  assert l == [(1, 3.14), (2, 3.14)], "error"
  l = list(gams.get('i', keyFormat=KeyFormat.TUPLE, valueFormat=ValueFormat.TUPLE, keyType=KeyType.INT))
  assert l == [((1,), ("text 1",)), ((2,), ("text 2",))], "error"
  l = list(gams.get('e2', keyType=KeyType.INT))
  assert l == [((1, 3), (3.14, 0, 3.14, 3.14, 1)), ((1, 4), (3.14, 0, 3.14, 3.14, 1))], "error"
$offEmbeddedCode

gams.set(symbolName, data, merge=False, domCheck=True)

This method sets the data for the GAMS symbol identified with symbolName. The parameter data takes a Python list or set containing items that represent the records of the symbol. Depending on the type and the dimension of the symbol, different formats can be used in order to specify the data. Different formats can not be mixed within one list. In general each record needs to be represented as a tuple containing the keys and the value field(s). Keys and/or values can also be enclosed in a tuple. Keys can be entered as labels (string) or label indexes (int). Value fields depend on the type of the symbol:

  • Parameters: One numerical value
  • Sets: explanatory text (optional)
  • Variable/Equations: Five numerical values: level, marginal, lower bound, upper bound, scale/prior/stage

IEEE special values float('nan'), float('inf'), and float('-inf') will be remapped to GAMS special values NA, INF, and -INF. The small numeric value 4.94066E-300 will be mapped into GAMS special value EPS.

The following Python code gives some examples on different valid formats for different symbol types and dimensions:

$onEmbeddedCode Python:
  # scalar parameter
  data = [3.14]
  data = [(3.14,)]

  # one dimensional parameters:
  data = [("i1", 3.14), ("i2", 3.14)]
  data = [(("i1",), 3.14), (("i2",), 3.14)]
  data = [("i1", (3.14,)), ("i2", (3.14,))]
  data = [(("i1",), (3.14,)), (("i2",), (3.14,))]

  # two dimensional parameter:
  data = [('i1', 'j1', 3.14), ('i1', 'j2', 3.14)]
  data = [(('i1', 'j1'), (3.14,)), (('i1', 'j2'), (3.14,))]

  # one dimensional sets:
  data = ['i1', 'i2']
  data = [('i1',), ('i2',)]

  # one dimensional sets with explanatory text
  data = [('i1', "text 1"), ('i2', "text 2")]
  data = [(('i1',), ("text 1",)), (('i2',), ("text 2",))]

  # scalar variables/equations
  data = [(3.14, 0, 0, 10, 1)]

  # one dimensional variables/equations:
  data = [("i1", 3.14, 0, 0, 10, 1), ("i2", 3.14, 0, 0, 10, 1)]
  data = [("i1", (3.14, 0, 0, 10, 1)), ("i2", (3.14, 0, 0, 10, 1))]
  data = [(("i1",), (3.14, 0, 0, 10, 1)), (("i2",), (3.14, 0, 0, 10, 1))]

  # two dimensional variables/equations:
  data = [("i1", "j1", 3.14, 0, 0, 10, 1), ("i1", "j2", 3.14, 0, 0, 10, 1)]
  data = [(("i1", "j1"), (3.14, 0, 0, 10, 1)), (("i1", "j2"), (3.14, 0, 0, 10, 1))]

  # using label indexes instead of labels
  data = [((1,), (3.14,)), ((2,), (3.14,))] # one dimensional parameter
  data = [((1,), ("text 1",)), ((2,), ("text 2",))] # one dimensional set
  data = [((1, 3), (3.14, 0, 0, 10, 1)), ((1, 4), (3.14, 0, 0, 10, 1))] # two dimensional equation/variable
$offEmbeddedCode

The optional parameter merge specifies if data in a GAMS symbol is merged (True) or replaced (False). The optional parameter domCheck specifies if Domain Checking is applied. Possible values are True and False.

Note
When calling gams.set() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.

gams.getUel(idx)

Returns the label corresponding to the label index idx

gams.mergeUel(label)

Adds label to the GAMS universe if it was unknown and returns the corresponding label index.

Note
When calling gams.mergeUel() in an embedded code section during execution time, new labels that are not known to the current GAMS program can not be added. The attempt will result in an execution error.

gams.getUelCount()

Returns the number of labels.

gams.printLog(msg)

Print msg to log.

gams.arguments

Contains the command line that was passed to the Python interpreter of the embedded code section. The syntax for passing arguments to the Python interpreter can be seen above.

gams.epsAsZero

Flag to read GAMS EPS as 0 (True) or as a small number, 4.94066E-300, when set to False. Default is True.

gams._debug

Debug flag that can be set to 1 for debug output. Default is 0.

For more examples on how to use the interface in Python see the following examples and tests:

Exchange via Files and Environment Variables

The Python class ECGamsDatabase provides read and write access to GAMS symbols. There are two other communication methods that can be used at GAMS compile time: files and environment variables. At compile time the Python code can produce a text file that can be included into GAMS via $include as in the following example:

$onEmbeddedCode Python: 10
  f = open('i.txt', 'w')
  for i in range(int(gams.arguments)):
    f.write('i'+str(i+1)+'\n')
  f.close()
$offEmbeddedCode

Set i /
$include i.txt
/;
display i;

Here the Python code received the number of elements to write to a text file via the argument after Python:. This text file is then included in the data statement of the GAMS set i. The display in the listing file looks as follows:

----     20 SET i  
i1 ,    i2 ,    i3 ,    i4 ,    i5 ,    i6 ,    i7 ,    i8 ,    i9 ,    i10

Python provides many packages to read input files for many different formats and hence can be used to transform such formats to a GAMS compatible input format, as an alternative to providing the data via list objects and the gams.set functionality.

The second alternative to exchange information at compile time are environment variables. GAMS and Python allow to get and set environment variables and hence can be conveniently used to exchange small pieces of information. The following code provides an example where the maximum value of a parameter b is needed to build a set k:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;

$onEmbeddedCode Python:
  import os
  kmax = int(max([b[1] for b in list(gams.get("b"))]))
  gams.printLog('max value in b is ' + str(kmax))
  os.environ["MAXB"] = str(kmax)
$offEmbeddedCode

$if x%sysEnv.MAXB%==x $abort MAXB is not set
Set k "from 0 to max(b)" / k0*k%sysEnv.MAXB% /;
Scalar card_k; 
card_k = card(k); 
Display card_k;

Alternatively in this example, we could build the GAMS set k in Python and send to GAMS via gams.set:

Set       i    / i1*i5 /;
Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /;
Set k "from 0 to max(b)" / system.empty /;

$onEmbeddedCode Python:
  kmax = int(max([b[1] for b in list(gams.get("b"))]))
  gams.printLog('max value in b is ' + str(kmax))
  gams.set("k",list(map(lambda k: 'k'+str(k), range(kmax + 1))))
$offEmbeddedCode k

Scalar card_k; 
card_k = card(k); 
Display card_k;

In both cases the resulting GAMS symbol k is the same and the display in the listing file looks as follows:

----     10 PARAMETER card_k               =       60.000  

Multiple Independent Python Sessions

At execution time the user has the ability to pause and continue an embedded code segment. Besides some performance aspects this also allows to work with multiple independent Python sessions. Due to some Python module incompatibilities (e.g. numpy) the independent Python sessions have to be enabled with a command line option pyMultInst set to 1. After the pauseEmbeddedCode we can obtain and store the handle of the last embedded code execution via function embeddedHandle. The handle needs to be supplied when we continue the Python session via continueEmbeddedCode. The different Python sessions are fairly separate as shown in the example below. Here we save the GAMS scalar ord_i in a Python object i five times. The value for i that we store in the five different Python sessions is 1 to 5. In the subsequent loop we activate the Python session with the appropriate handle and print the value of i:

$if not %sysEnv.GMSPYTHONMULTINST%==1 $abort.noError Start with command line option pyMultInst=1
Set       i     / i1*i5 /; 
Parameter h(i) 
          ord_i / 0 /;
loop(i,
  ord_i = ord(i);
  embeddedCode Python:
    i = int(list(gams.get("ord_i"))[0])
    gams.printLog(str(i))
  pauseEmbeddedCode
  h(i) = embeddedHandle;
);
loop(i,
  continueEmbeddedCode h(i):
    gams.printLog(str(i))
  endEmbeddedCode
);

The GAMS log shows the value of i in the different Python sessions:

--- Starting execution: elapsed 0:00:00.002
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 1
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 2
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 3
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 4
--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- 5
--- Execute embedded library embpycclib64.dll
--- 1
--- Execute embedded library embpycclib64.dll
--- 2
--- Execute embedded library embpycclib64.dll
--- 3
--- Execute embedded library embpycclib64.dll
--- 4
--- Execute embedded library embpycclib64.dll
--- 5
*** Status: Normal completion

Troubleshooting Embedded Python Code

The GAMS compiler ensures that the number of errors during execution time is minimized. While the logic of the GAMS program might be flawed there is nothing (with a few exceptions) that the GAMS system cannot execute. This is different if we embed foreign code in a GAMS program. The GAMS compiler does not understand the foreign code syntax and just skips over it. Only when the code is executed we will find out if everything works as expected. If the embedded code contains some (Python) syntax errors the Python parser will inform us about this and the message will appear in the GAMS log. For example, the following Python code using the gams.printLog function two times will generate a syntax error:

$onEmbeddedCode Python:
  gams.printLog('hello')
   gams.printLog('world...')
$offEmbeddedCode

The GAMS log will provide some guidance:

--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll File "C:\tmp\gamsdir\225a\myPy.dat.dat", line 8
    gams.printLog('world...')
    ^
IndentationError: unexpected indent

--- Python error! Return code from Python execution: -1
*** Error executing Python embedded code section:
*** Check log above

Moreover, if the Python code raises an exception which is not handled within the code this will also lead to a compilation or execution error in GAMS depending at what phase the embedded code is executed.

embeddedCode Python:
  raise Exception('something is wrong')
endEmbeddedCode

will produce the following GAMS log and an execution time error:

--- Initialize embedded library embpycclib64.dll
--- Execute embedded library embpycclib64.dll
--- Exception from Python: something is wrong
*** Error at line 1: Error executing "embeddedCode" section: Check log above

The Python code is executed as part of the GAMS process and GAMS gives control to the Python interpreter when executing the embedded code. So in the worst case if the Python interpreter crashes, the entire GAMS process will crash. Therefore, it is important to be able to test and debug the embedded Python code independent of GAMS. In the following examples we call the Python interpreter executable as part of a GAMS job. In principle this can be tested and debugged completely independent of GAMS where a GDX file represents the content of the GAMS database.

In the first example we mimic the embedded code facility at compile time by exporting the entire GAMS database to a GDX file debug.gdx. With $on/offEcho we write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block and execute the Python interpreter via $call. One of the extra lines at the end of the embedded code triggers the creation of a GDX result file debugOut.gdx which can be imported in subsequent $gdxin/$load commands.

Set i /i1*i10/
    p(i,i) permutation;

$gdxOut debug.gdx
$unload
$gdxOut

$onEcho > debug.py
from gamsemb import *
gams = ECGAMSDatabase('debug.gdx')
try:
  import random
  i = list(gams.get("i"))
  p = list(i)
  random.shuffle(p)
  for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
  gams.set("p", p)
  gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
  print(str(e))
$offEcho

$call ="%gams.sysdir%GMSPython/python" debug.py
$if errorlevel 1 $abort Problems running Python

$gdxIn debugOut.gdx
$loadDC p

Option p:0:0:1;
Display p;

In the second example we mimic the embedded code facility at execution time by exporting the entire GAMS database to a GDX file debug.gdx via execute_unload 'debug.gdx';. We write the embedded code with a few extra lines at the top and bottom surrounded by a try/except block via the put facility and execute the Python interpreter via execute. Identical to the compile time example, we export the result to the GDX file debugOut.gdx which can be imported via the execute_load statement.

Set i /i1*i10/
    p(i,i) permutation;

execute_unload 'debug.gdx';
file fpy / 'debug.py' /; put fpy;

$onPut
from gamsemb import *
gams = ECGAMSDatabase('debug.gdx')
gams.arguments = '-a c -b -db abc'
try:
 import random
 i = list(gams.get("i"))
 p = list(i)
 random.shuffle(p)
 for idx in range(len(i)):
     p[idx] = (i[idx], p[idx])
 gams.set("p", p)
 gmdWriteGDX(gams._gmd,'debugOut.gdx',1);
except Exception as e:
 print(str(e))
$offPut

putClose fpy;
execute '="%gams.sysdir%GMSPython/python" debug.py';
abort$errorlevel 'problems running python';

execute_load 'debugOut.gdx', p;
Option p:0:0:1;
Display p;

Performance Considerations of Embedded Python Code

If the same embedded code section (e.g. in a loop) is executed many times there are a few considerations to be taken into account in order to get the best performance. For this we will experiment with the example from the introduction. We look for a random permutation of a set i. In addition we have a cost matrix c(i,ii) and we are looking for the least cost permutation. We should just formulate this as matching in a bi-partite graph but in order to demonstrate some performance considerations we will repeatedly call the Python code that provides a random permutation and we will evaluate the cost of the permutation in GAMS and store the value of the cheapest one. Here is the naive implementation using embedded code:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation; 
c(i,ii) = uniform(-50,50);

Set    iter / 1*100 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  embeddedCode Python:
    import random
    i = list(gams.get("i"))
    p = list(i)
    random.shuffle(p)
    for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
    gams.set("p", p)
  endEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
Display minTCost;

In the code we start and stop the Python interpreter and with every execution of the embedded code we need to make the setup and initialization which takes a significant amount of time. The entire GAMS job executes in about 16 seconds. We can avoid the repeated setup and initialization by using pause and continue:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation;
c(i,ii) = uniform(-50,50);

embeddedCode Python:
  import random
pauseEmbeddedCode

Set    iter / 1*1000 /;
Scalar tcost
       minTCost / +inf /;
loop(iter,
  continueEmbeddedCode:
    i = list(gams.get("i"))
    p = list(i)
    random.shuffle(p)
    for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
    gams.set("p", p)
  pauseEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
continueEmbeddedCode:
  pass
endEmbeddedCode
Display minTCost;

The last embedded code execution of the Python pass statement is to clean up and terminate the Python session. As you can see from the set iter we had to increase this from 100 to 1000 to measure the timing properly. This run takes about 1.179 secs (we ran this 20 times and build the average). This is the biggest improvement, the other two following enhancements are just icing on the cake. We can actually extract the set i and store this in Python list i just once:

Set i / i1*i50 /
    p(i,i) permutation; 
Alias (i,ii);
Parameter c(i,i) cost of permutation; 
c(i,ii) = uniform(-50,50);

embeddedCode Python:
  import random
  i = list(gams.get("i"))
pauseEmbeddedCode

set    iter / 1*1000 /;
scalar tcost
       minTCost / +inf /;
loop(iter,
  continueEmbeddedCode:
    p = list(i)
    random.shuffle(p)
    for idx in range(len(i)):
      p[idx] = (i[idx], p[idx])
    gams.set("p", p)
  pauseEmbeddedCode p
  tcost = sum(p, c(p));
  if (tcost < minTCost, minTCost = tcost);
);
continueEmbeddedCode:
pass
endEmbeddedCode
Display minTCost;

The total running time of this is 1.005 secs. In addition, we can work with label indexes rather than the labels itself. Indexes are integers and are often faster than labels that are stored as strings. The only difference to the code above is the extraction method of the Python list i by i = list(gams.get("i",keyType=KeyType.INT)). The resulting running time is 0.993 secs.

Porting to a Different Version of Python

GAMS comes with a Python 3.6 interpreter that is used as default interpreter in the Embedded Code Facility. It is located in [GAMS directory]/GMSPython and comes with the usual tools for e.g. installing further Python modules and packages like pip and easy_install. These tools need to be executed in the directory in which they are located since they are referencing the Python interpreter using a relative path. In order to make them work from arbitrary locations as well one can either change the so-called Shebang (#!) line manually or run the script fixPath.py which does the work for a list of specific programs.

cd [GAMS directory]/GMSPython/bin
./python3.6 fixPath.py

The Embedded Code Facility using the Python programming language is implemented for Python 3.6. Instead of using the default Python interpreter that comes with GAMS, it is possible to use another Python installation instead. This can be done by setting the command line option pySetup to 0. Since this only turns off the use of the default Python version, one still needs to make the system aware of an alternative Python installation. When using the Embedded Code Facility a version specific library of Python is loaded (e.g. Python36.dll). The location and name depends on the Python version and the operation system that is used. One needs to make sure that this library is found by the system. One way to achieve this is by adding the location of the library to the PATH environment variable (Window), the LD_LIBRARY_PATH environment variable (Linux) or the DYLD_LIBRARY_PATH environment variable (Mac OS X). Furthermore, one might need to set the PYTHONHOME environment variable to the alternative Python installation. In order for the Embedded Code Facility to work, one needs to make the alternative Python interpreter aware of the required modules provided by GAMS. This can be done by installing them or setting PYTHONPATH appropriately. See the Getting started of the GAMS Python API for further instructions on installing the Python API. Note that the supported Python version for embedded code is Python 3.6 and that beside the GAMS Python API, the module gamsemb.py is required which is located in [GAMS directory]/apifiles/Python/api_36 only. If the setup.py script is used for installation, gamsemb.py will only be installed if the Python version is 3.6. Note that if a different version of Python is used in an experimental setup, it might be required to build the embedded code library manually. How to do this is explained in the paragraph below.

Although the Embedded Code Facility and its binary components are part of the GAMS distribution, it is possible to build it manually from source using the following commands. The exact command line might change depending on the compiler and the operating system in use:

  • Windows (32 bit):
    cd [GAMS directory]/apifiles/C/api
    cl.exe -Feembpycclib.dll -I../../../GMSPython/include -I. embpyoo.c emblib.c gmdcc.c ../../../GMSPython/libs/python36.lib -LD -link -def:emblib.def
  • Windows (64 bit):
    cd [GAMS directory]/apifiles/C/api
    icl.exe -Feembpycclib64.dll -I../../../GMSPython/include -I. embpyoo.c emblib.c gmdcc.c ../../../GMSPython/libs/python36.lib -LD -link
  • Linux (64 bit):
    cd [GAMS directory]/apifiles/C/api
    gcc -o libembpycclib64.so -nostartfiles -shared -Wl,-Bsymbolic -m64 -pthread -Wl,-rpath,\$ORIGIN -Wl,-rpath,\$ORIGIN/GMSPython/lib -fPIC -I ../../../GMSPython/include/python3.6m -I. embpyoo.c emblib.c gmdcc.c -L../../../GMSPython/lib/ -lpython3.6m -ldl
  • Mac OS X:
    cd [GAMS directory]/apifiles/C/api
    gcc -o libembpycclib64.dylib -dynamiclib -m64 -shared -Wl,-rpath,@loader_path/. -Wl,-rpath,@loader_path/GMSPython/lib -fPIC -I ../../../GMSPython/include/python3.6m -I. embpyoo.c emblib.c gmdcc.c -L../../../GMSPython/lib/ -lpython3.6m -ldl