Table of Contents
- Note
- This feature is currently in beta status.
Introduction
GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.
Getting Started
The first step in getting started with GAMS Jupyter Notebooks is to make your Python 3 installation aware of the GAMS Python API collection described in the Getting started section of the API tutorial. We recommend to follow the steps below which are specifically tailored for getting started with GAMS Jupyter notebooks. While any Python >=3.6 installation is supported, we recommend the use of Anaconda or Miniconda Python distributions.
In addition to the GAMS Python API collection, the packages jupyterlab, matplotlib, pandas, and tabulate are required in order to enable GAMS Jupyter Notebooks and to run the provided examples which are located in apifiles/Python/jupyter_examples. The following code section shows how to create and set up a Miniconda Python environment for GAMS Jupyter notebooks:
Windows:
conda create -n gmsjupyter python=3.8 jupyterlab pandas tabulate matplotlib conda activate gmsjupyter cd C:\GAMS\33\apifiles\Python\api_38 python setup.py install (In case of permission problems run the following instead: python setup.py build -b %TEMP%\build install) cd C:\jupyter (or another directory that should be used for your notebooks) jupyter notebook (this will start the jupyter notebook server)
Linux:
conda create -n gmsjupyter python=3.8 jupyterlab pandas tabulate matplotlib conda activate gmsjupyter cd ~/gams/gams33.1_linux_x64_64_sfx/apifiles/Python/api_38 python setup.py install (In case of permission problems run the following instead: python setup.py build -b $TMPDIR/build install) cd ~/jupyter (or another directory that should be used for your notebooks) jupyter notebook (this will start the jupyter notebook server)
macOS:
conda create -n gmsjupyter python=3.8 jupyterlab pandas tabulate matplotlib conda activate gmsjupyter python /Library/Frameworks/GAMS.framework/Resources/apifiles/Python/api_38/setup.py build -b $TMPDIR/build install cd ~/jupyter (or another directory that should be used for your notebooks) jupyter notebook (this will start the jupyter notebook server)
The notebooks Millco.ipynb and Introduction.ipynb located in apifiles/Python/jupyter_examples are good starting points to get familiar with Jupyter notebooks and GAMS. The Tutorial in this documentation is derived from the Introduction.ipynb notebook.
Tutorial
GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.
The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to load the extension gams_magic:
In:
There are a few other useful command in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas dataframes useful for exchange with GAMS have been collected in the DataTransform.ipynb. The next cell will execute that notebook and make such data transformation function, e.g. gt_from2dim (see below) available in this notebook. %%capture captures the output from the execution of the notebook and does not clutter your output.
In:
Somehow one output from a cell is sometimes not enough, e.g. if you want to display a couple of tables. The display function allows you to do this. As an example, we display two Python lists:
In:
[1, 2, 3]
[4, 5, 6]
Running GAMS code
Running GAMS code can be done by using either %gams (line magic) or %%gams (cell magic). While %gams can be used for running a single line of GAMS code, %%gams makes the whole cell a GAMS cell. While %gams can appear on any line, %%gams is only valid in the first line of a cell.
In:
In:
The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple time, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.
In:
In:
You won't see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log and %gams_lst to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e only display the section of the listing file associated with the execution:
In:
E x e c u t i o n
---- 11 PARAMETER p2
j1 j2 j3 j4 j5
i1 1.000 1.000 1.000 1.000 1.000
i2 1.000 1.000 1.000 1.000 1.000
i3 1.000 1.000 1.000 1.000 1.000
i4 1.000 1.000 1.000 1.000 1.000
i5 1.000 1.000 1.000 1.000 1.000
There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset. For example, since we declared parameter p2 already over (i,j) we cannot change our mind and redeclare p2 as parameter p2(i,i,j):
In:
This will give you a compilation error and an exception in the cell execution (uncomment the line in the next cell to do so):
In:
With a %gams_reset we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too. The state of the GAMS database is kept in various files that can easily clutter your directory. The %gams_cleanup call helps you to clean the directory of temporary files. The option -k keeps the most recent GAMS database, hence the %gams_cleanup -k is a save call anywhere in your notebook.
In:
Pushing Data from Python to GAMS
%gams_push transfers data from Python to GAMS. Supported data types for pushing data are lists, pandas.DataFrame and numpy arrays:
In:
As mentioned above the execution of a %%gams cell or %gams and %gams_push line magic does not produce output. If one wants to verify that the data ended up in GAMS we can display the symbols in GAMS and output the corresponding part of the listing file:
In:
E x e c u t i o n
---- 12 SET i
i1, i2, i3
---- 12 SET j
j1, j2
---- 12 PARAMETER p
j1 j2
i1 1.100 2.200
i2 3.300 4.400
i3 5.500 6.600
The next cell turns a Python list into a pandas.DataFrame, multiplies the value by 2 and displays the dataframe with IPythons's display. We actually display the transformed p2 (via function gt_pivot2d found in the DataTransformation notebook run at the top of the notebook), so the table looks nicer. Next, we sends the pandas.DataFrame down to GAMS via the %gams_push command. Via the GAMS display and the output of the relevant part of the listing file we see that the %gams_push succeeded:
In:
| j1 | j2 | |
|---|---|---|
| i1 | 2.2 | 4.4 |
| i2 | 6.6 | 8.8 |
| i3 | 11.0 | 13.2 |
E x e c u t i o n
---- 18 PARAMETER pp
j1 j2
i1 2.200 4.400
i2 6.600 8.800
i3 11.000 13.200
When using numpy arrays in order to push data into GAMS, the data is assumed to be dense. The correspondng sets are defined automatically from 1..n, 1..m, etc depending on the data that is pushed.
In:
In:
E x e c u t i o n
---- 23 SET i
1, 2, 3
---- 23 SET j
1, 2
---- 23 SET k
1, 2
---- 23 PARAMETER p3 3-dim Matrix
1 2
1.1 1.100 -1.100
1.2 2.200 -2.200
2.1 3.300 -3.300
2.2 4.400 -4.400
3.1 5.500 -5.500
3.2 6.600 -6.600
Pulling Data from GAMS to Python
The line magic %gams_pull transfers data from GAMS to Python in different formats. Supported formats are lists (default), pandas.DataFrame and numpy arrays. The following example pulls the sets i, j, and parameter p3 from GAMS into lists. For multi-dimensional symbols the records become Python tuples. Currently, the renaming functionality %gams_pull gamsSym=pySymbol is not yet supported.
In:
['1', '2', '3']
['1', '2']
[('1', '1', '1', 1.1),
('1', '1', '2', -1.1),
('1', '2', '1', 2.2),
('1', '2', '2', -2.2),
('2', '1', '1', 3.3),
('2', '1', '2', -3.3),
('2', '2', '1', 4.4),
('2', '2', '2', -4.4),
('3', '1', '1', 5.5),
('3', '1', '2', -5.5),
('3', '2', '1', 6.6),
('3', '2', '2', -6.6)]
The switch -d will populate pandas.DataFrames instead of lists with the GAMS data. The dataframes that are pushed into or pulled from GAMS have a very specific layout. There is a record index and the GAMS domains show up as columns in the dataframe. For parameters, there is an extra value column. For variables and equations we find extra columns level, marginal, lower, upper, and scale. The method head() used in the IPython display provides only the first 5 records of a pandas.DataFrame:
In:
| i | |
|---|---|
| 0 | 1 |
| 1 | 2 |
| 2 | 3 |
| j | |
|---|---|
| 0 | 1 |
| 1 | 2 |
| i | j | k | value | |
|---|---|---|---|---|
| 0 | 1 | 1 | 1 | 1.1 |
| 1 | 1 | 1 | 2 | -1.1 |
| 2 | 1 | 2 | 1 | 2.2 |
| 3 | 1 | 2 | 2 | -2.2 |
| 4 | 2 | 1 | 1 | 3.3 |
| i | level | marginal | lower | upper | scale | |
|---|---|---|---|---|---|---|
| 0 | 1 | 1.0 | 0.0 | -inf | inf | 1.0 |
| 1 | 2 | 0.0 | 3.0 | -inf | inf | 1.0 |
The data transformation functions available from DataTransformations.ipynb help to convert between this format and formats more suitable for display of other transformations in Python. The following lines give a quick overview of the transformation functionality:
In:
| i | j | value | |
|---|---|---|---|
| 0 | 1 | 1 | 2.0 |
| 1 | 1 | 2 | 9.0 |
| 2 | 2 | 1 | 6.0 |
| 3 | 2 | 2 | 4.0 |
| 4 | 3 | 1 | 3.0 |
| 5 | 3 | 2 | 3.0 |
| 1 | 2 | |
|---|---|---|
| 0 | 2.0 | 9.0 |
| 1 | 6.0 | 4.0 |
| 2 | 3.0 | 3.0 |
| i | j | value | |
|---|---|---|---|
| 0 | 1 | 1 | 2.0 |
| 1 | 1 | 2 | 9.0 |
| 2 | 2 | 1 | 6.0 |
| 3 | 2 | 2 | 4.0 |
| 4 | 3 | 1 | 3.0 |
| 5 | 3 | 2 | 3.0 |
The switch -n will populate numpy arrays instead lists with the GAMS parameters. This format works with parameters only! The GAMS data will be dropped into a dense numpy array:
In:
array([[1., 0.],
[0., 2.],
[0., 0.]])
Troubleshooting and Hints
- Paths to notebooks must not contain whitespaces. A notebook file itself (*.ipynb) can.
- The directory containing the notebook will clutter up with temporary files. A
%gams_cleanupor%gams_cleanup -kif you want to continue working in this session - The temporary file are useful for debugging, see below. The naming of the temporary files is not very sophisticated, so it can come to file nameing conflicts if you run two notebooks in the same directory at the same time (in different browser tabs). Create subdirectories and move the notebook into the subdirectories if you run into this problem.
- As soon as en error occurs while running GAMS code (the notebook exception is a GamsExecption), it can be useful to examine the listing file (*.lst) using
%cat path/to/listing.lstor%gams_lst. The path of the listing file can be found in the last line of the output of a failing cell.
In:
---------------------------------------------------------------------------
GamsExceptionExecution Traceback (most recent call last)
/tmp/ipykernel_24224/1159222853.py in <module>
----> 1 get_ipython().run_line_magic('gams', 'Parameter p5(i,j,l);')
/opt/miniconda3/envs/JupyterHub/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
2349 kwargs['local_ns'] = self.get_local_scope(stack_depth)
2350 with self.builtin_trap:
-> 2351 result = fn(*args, **kwargs)
2352 return result
2353
/opt/miniconda3/envs/JupyterHub/lib/python3.8/site-packages/decorator.py in fun(*args, **kw)
230 if not kwsyntax:
231 args, kw = fix(args, kw, sig)
--> 232 return caller(func, *(extras + args), **kw)
233 fun.__name__ = func.__name__
234 fun.__doc__ = func.__doc__
/opt/miniconda3/envs/JupyterHub/lib/python3.8/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):
/opt/miniconda3/envs/JupyterHub/lib/python3.8/site-packages/gams_magic/gams_magic.py in gams(self, line, cell)
489 opt.traceopt = 3
490 with open(jobName + ".log", "w") as logFile:
--> 491 self.job.run(opt, checkpoint=self.cp, output=logFile)
492 solveSummary = self.parseTraceFile(trcFilePath)
493
/opt/miniconda3/envs/JupyterHub/lib/python3.8/site-packages/gams/execution.py in run(self, gams_options, checkpoint, output, create_out_db, databases)
905 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), set the debug flag of the GamsWorkspace constructor to DebugLevel.KeepFiles or higher or define a working_directory to receive a listing file with more details", exitcode)
906 else:
--> 907 raise gams.workspace.GamsExceptionExecution("GAMS return code not 0 (" + str(exitcode) + "), check " + self._workspace._working_directory + os.path.sep + tmp_opt.output + " for more details", exitcode)
908 self._p = None
909
GamsExceptionExecution: GAMS return code not 0 (2), check /home/share/jupyter_examples/gamsJupyter16.lst for more details
In:
GAMS 37.1.0 r07954d5 Released Nov 11, 2021 LEX-LEG x86 64bit/Linux - 01/20/22 09:02:26 Page 26
G e n e r a l A l g e b r a i c M o d e l i n g S y s t e m
C o m p i l a t i o n
46 Parameter p5(i,j,l);
**** $120
**** LINE 3 INPUT /home/share/jupyter_examples/gamsJupyter16.gms
**** 120 Unknown identifier entered as set
**** 1 ERROR(S) 0 WARNING(S)
COMPILATION TIME = 0.001 SECONDS 3 MB 37.1.0 r07954d5 LEX-LEG
**** FILE SUMMARY
Restart /home/share/jupyter_examples/_gams_py_gcp0.g00
Input /home/share/jupyter_examples/gamsJupyter16.gms
Output /home/share/jupyter_examples/gamsJupyter16.lst
Save /home/share/jupyter_examples/_gams_py_gcp13.g0?
**** USER ERROR(S) ENCOUNTERED