Embedding Trained Neural Networks in GAMSPy

Presented by: Burak Usul

GAMSPy is a powerful mathematical optimization package that combines Python’s flexibility with GAMS’s modeling performance. GAMSPy enables previously challenging applications in the area of combining machine learning (ML) and mathematical modeling. To support these ML applications, our work introduces essential ML operations into GAMSPy, such as matrix multiplication, transposition, and norm calculations. Building on this foundation, we introduce GAMSPy "formulations", a straightforward way to model common neural network constructs like linear (dense) layers, convolutional layers, and activation functions (ReLU, tanh and so on). When there are many good ways to formulate a construct, we implement many of them and to let user decide depending on their use case. In addition to neural network construct, we also introduce formulations for classical ML constructs such as regression trees and so on. In this talk, we demonstrate these enhancements by generating adversarial images for the German Traffic Sign Recognition Benchmark (GTSRB) using GAMSPy. We selected GTSRB because it requires a neural network that is significantly larger than many other toy examples neural networks like the ones trained for MNIST. Recognizing the challenges of embedding large-scale neural networks, we also propose a method for incorporating large models via black-box embeddings. We highlight our new black-box formulation and its complementary formulations, emphasizing their potential in ML research and development.