Update GAMS to version 45.4.0.

Breaking changes

• The nginx logs now include the request length as an additional parameter. For a full example, see here.

New features

• Update GAMS to version 45.3.0.

New features

• Update GAMS to version 45.2.0.
• Engine now allows admins to set a password policy for all the users who use Engine as their identity provider. When new users register, they must choose a password that complies with the password policy. Already registered users are not effected. The minimum length can be specified in the password policy. It is also possible to specify whether an uppercase letter, a lowercase letter, a number or a special character must be included. It is also possible to disallow frequently used passwords.
• Update Python version of worker from 3.7 to 3.11.

API Changes

• New password policy feature was added.
• The following endpoints are added to the /auth namespace for the new password policy feature:
  • PUT /auth/password-policy and GET /auth/password-policy
• PUT /auth/password-policy is used to update the password policy. Requires admin role.
• GET /auth/password-policy is used to retrieve the current password policy. Requires login.

Bug fixes

• Fix an issue where updating the inviteable identity providers of an inviter whose invitee was deleted caused problems.
• Fix an issue where adding/deleting user groups could cause an Internal Error rather than Forbidden.

Engine UI

• The Administration section now inlcudes a button to update the password policy.
• When a new user registers, an information icon now provides the current password policy.

New features

• Use a new parser for multipart body requests to endpoints: POST /hypercube/, POST/PATCH /namespaces/{namespace}/models/{model}, and POST /jobs/ for faster file upload times.

Bug fixes

• Fix race condition where waiting for child processes could crash the worker.
• Add missing globbing_enabled field to the response of the GET /namespaces/{namespace} endpoint for listing models.

Engine UI

• Display globbing_enabled field while showing, creating or updating models.

Update GAMS to version 44.4.0.

Update GAMS to version 44.3.0.

New features

• Increased default worker interrupt listener wait timeout to 10 seconds and introduced environment variable GMS_RUNNER_WORKER_SIGINT_LISTENER_TIMEOUT to change it, which defaults to 10 if not provided.
• Workers will no longer send empty log messages to avoid unnecessary load on the queue.
• Engine SaaS: API now gently closes database connections for long-running requests when not needed to improve scalability.

Bug fixes

• Fix issue where a job's resources were not cleaned up when the job failed to start due to out of quota.
• Fix minor security issue due to use of static temporary file paths.
• Fix race condition where cancelled or waiting jobs could be marked as cancelled if their namespace was deleted.
• Fix race condition where hypercube results were uploaded even when the namespace was deleted.
• Fix race condition where if a job is cancelled before GAMS starts, the kill signal could be ignored and GAMS started.
• Fix issue where the times array in the GET /usage endpoint was not always sorted from the earliest to the latest solution attempt.

API Changes

• INEX files now accept the globbing_enabled boolean field. It is optional and defaults to true. If you know the exact filenames, you can set it to false and any filename you list will be considered exact. Example:
  { "type": "exclude", "files": ["trnsport.gms"], "globbing_enabled": false }
• Now stdout_filename is marked as nullable in the job model. Engine already allowed stdout_filename to be set to null when sending a job with a JSON request. This change is to complete this feature so that auto-generated clients do not complain.

Bug fixes

• Engine SaaS: Fix bug where the name of the instance assigned to a pool was reported in the GET /usage/ endpoint, not the name of the pool itself when a pool was used as the default instance.

Bug fixes

• Fix bug in the Engine UI that could prevent users from uploading files when submitting a job through Google Chrome.
• Performance and memory improvements in Cleaner.

All of our scripts now use Docker Compose version 2, as version 1 has reached its end of life.

Update GAMS to version 44.1.0.

Update GAMS to version 43.4.1.

Update GAMS to version 43.4.0.

Bug fixes

• Fix issue where changing default instance via the PUT /usage/instances/{username} endpoint did not work in certain situations.

API Changes

• The POST /auth/oauth2-providers and POST /auth/oidc-providers endpoints now accept a new field token_endpoint_auth_methods_supported that stores information about what authentication methods the token endpoint supports. The GET /auth/oauth2-providers, GET /auth/oidc-providers, GET /auth/providers and GET /auth/providers/all endpoints now return this field. In case client_secret_basic is not supported, the POST /auth/oauth2-token endpoint fall back to client_secret_post for authentication.

Bug fixes

• Fix JSON request parser for the POST/PUT /usage/instances endpoints for the fields: tolerations and node_selectors.

API Changes

• The maximum length of the web_ui_client_secret field in the POST/PUT /auth/oauth2-providers and POST/PUT /auth/oidc-providers endpoints as well as the device_client_secret field in the POST/PUT /auth/oidc-providers endpoints has been increased from 128 to 512 characters.

Update GAMS to version 43.3.1.

New features

• GET /users/ endpoint accepts a new query parameter filter when set to deleted=false or deleted=true can filter deleted or active users. For example GET /users/?filter=deleted%3dfalse to get only active users.
• GET /users/invitation endpoint accepts a new query parameter filter when set to used=false or used=true can filter used invitations. For example GET /users/invitation?filter=used%3dfalse to get non-used invitations.

Bug fixes

• Fix issue where Engine does not accept Engine generated access tokens when using identity providers of type OIDC.

Breaking Changes

• When registering a new webhook via the POST /users/webhooks, if no events are provided, an error code 400 BAD REQUEST is now issued as ALL events are no longer subscribed to by default.

New features

• Engine SaaS: Introducing instance pools: a number of workers with a homogeneous instance.
• Add new events for webhooks:
  • When a job runs out of resources (e.g. memory, disk space, volume quota etc.)
  • When a job is running for a certain time (set by the user)
  • When a user's volume quota reaches a certain threshold (set by the user)

API Changes

• The maximum length of the label of instances has been increased from 30 to 60 characters.

Update GAMS to version 43.3.0.

Bug fixes

• Fix issue in the worker that caused a database deadlock when the database was under heavy load.

Breaking Changes

The maximum number of files that can be cleaned with the DELETE /cleanup/results endpoint has been reduced from 10000 to 1000. The return code when exceeding this limit is now 413 instead of 400.

Update GAMS to version 42.5.0.

Update GAMS to version 42.4.0.

Bug fixes

• Fix issue in the Engine API that prevented changing an inviter's inviteable identity providers when they had unregistered invitations.
• Fix issue in the Engine UI that made it impossible to register a user with Engine as an identity provider.
• Fix issue in the Engine UI that prevented inviters with a single inviteable identity provider that is not Engine from inviting users.
• Fix issue in the Engine UI that caused inviteable identity providers to not update in certain situations.
• Fix Swagger UI not being displayed in browsers due to CSP configuration.

New features

• Update GAMS to version 42.3.0.
• External identity providers of type OpenID Connect, OIDC, are introduced. You can now can now delegate authentication to your OIDC identity provider. See documentation. Following endpoints are added:
  • GET /auth/oidc-providers
  • PUT /auth/oidc-providers
  • POST /auth/oidc-providers
  • POST /auth/oidc-providers/login

API Changes

• POST /auth/oauth2-token endpoint now also forwards tokens for OIDC providers. It also now supports token forwarding for device code flow. You can use the new grant_type field (defaults to authorization_code) to specify the type of flow. If "authorization_code" is used, the mandatory fields are the same, but if "urn:ietf:params:oauth:grant-type:device_code" is used, you need to set the new "device_code" field instead. For device code flow, you should not repeat your request unless the endpoint returns a 428 status code indicating that authorization is still pending.
• GET /auth/providers and GET /auth/providers/all endpoints have a new "oidc" field for listing OIDC-type identity providers.

Bug fixes

• Fix the integrity error from the forward proxy when the response contained white-space characters
• Fix issue with creation of access token when logging in via LDAP provider, where Engine did not set scopes for the access token.
• The job status is now changed from "Queued" to "Running" before data extraction and not afterwards.

Engine UI

• Move buttons to update Engine/GAMS licenses from Users to Administration section.

Update GAMS to version 42.2.0.

Bug fixes

• Restructured workers' database connections to get rid of idle transactions.
• Fix exceptions in worker logs when using stream entries and interrupting jobs.
• Fix some issues that could cause problems with installation on Windows.
• If no scope was specified at login, the API would provide all the scopes, including READONLY, which caused some clients not to work. The READONLY scope has been removed from the default scopes.

Update GAMS to version 42.1.0.

Breaking Changes

• When extracting ZIP files (both model data and data) on the worker, the file permissions are now preserved. This causes problems with MIRO versions < 2.2. Conversely, this means that this Engine version is only compatible with MIRO versions >= 2.2.

API Changes

• New status field was added to the hypercube_job_usage object of the GET /usage/ endpoint.

Bug fixes

• Fix issue with event manager hanging indefinitely when connection to queue is lost.
• Fix issue that could cause workers to not properly respond to the cancellation of a job in rare cases, even if hard_kill was set to true.

Bug fixes

• Important bug fix: RabbitMQ introduced a default value for delivery acknowledgement timeouts with a patch release, which caused any job longer than 30 minutes to fail. The migration provides an extended configuration file to not use the default value of 30 minutes, allowing longer jobs to run.

New features

• Engine now supports external identity providers. You can invite users and log in using LDAP and OAuth2 identity providers. For more information, see the identity provider documentation.
• For new installations of Engine, the JWT secret is generated lazily, rather than during the boot up. And JWT secrets are now more secure.

API Changes

• The CONFIGURATION and AUTH scopes have been added. The CONFIGURATION scope controls access to the PATCH /configuration endpoint. The AUTH scope controls access to each endpoint in the auth namespace.
• Now the inviters can change the passwords of the invitees via PUT /users/ endpoint.
• Made the password field for the POST /users/ optional. If an external identity provider is used, this field must be left blank. If Engine is the identity provider, this field must be specified.
• The GET /users/ endpoint now accepts everyone parameter. This parameter is optional and defaults to true if not specified. This allows using this endpoint to retrieve information about the logged in user by setting everyone to false.
• The endpoint GET /users/ now also returns identity_provider_name, identity_provider_user_subject.
• The fields identity_provider_name, identity_provider_user_subject, and invitable_identity_providers are added to POST /users/invitation endpoint. If the inviting user can invite with multiple identity providers or has a single identity provider that is not Engine, these fields must be filled in. invitable_identity_providers is relevant only if inviter role is attached to the invitation.
• The endpoint GET /users/invitation now also returns identity_provider_name, identity_provider_user_subject and invitable_identity_providers.
• hostname parameter added to PATCH /configuration endpoint. GET /configuration displays hostname as well. See Identity Provider documentation to see its usage.
• scope parameter added to endpoint POST /auth/ to control the scopes in the generated JWT. Users who use an external identity provider cannot request an AUTH scope.
• Added grant_type and scope parameters to enpoint POST /auth/login to follow Resource Owner Password Credentials grant from RFC6749. However, the grant_type parameter is optional for backward compatibility. The preferred method for providing scopes is now to use scope parameter instead of access_scopes. The former accepts space-separated scopes, while the latter accepts an array of scopes.
• The following endpoints are added to auth namespace for the new identity provider feature:
  • GET /auth/providers and DELETE /auth/providers
  • GET /auth/providers/all
  • GET /auth/oauth2-providers, POST /auth/oauth2-providers, PUT /auth/oauth2-providers
  • POST /auth/oauth2-token
  • GET /auth/ldap-providers, POST /auth/ldap-providers, PUT /auth/ldap-providers
  • POST /auth/ldap-providers/<string:provider_name>/login
  The following endpoints are added to users namespace for the new identity provider feature:
  • PUT /users/identity-provider
  • PUT /users/inviters-providers/<string:username> and GET /users/inviters-providers/<string:username>
  • GET /users/invitation/<string:token>
  This section only summarizes their usage. For a more detailed explanation, see the identity provider's documentation.
• GET /auth/providers endpoint does not require authentication and can be used to list non-hidden identity providers. This endpoint also accepts the name parameter, if specified, only the specified identity provider will be displayed, regardless of whether it is hidden or not.
• DELETE /auth/providers is used to delete identity providers. Requires admin role.
• GET /auth/providers/all is used to list all identity providers, whether they are hidden or not. Requires admin role.
• GET /auth/oauth2-providers is used to list all the OAuth2 identity providers. Requires admin role as it displays web_ui_client_secret. Users can access the information of the OAuth2 providers except secrets via GET /auth/providers.
• POST /auth/oauth2-providers is used to create an OAuth2 identity provider. Requires admin role. If web_ui_client_secret is specified, it is stored encrypted.
• PUT /auth/oauth2-providers is used to update an OAuth2 identity provider. Requires admin role.
• Engine UI is a single-page application and therefore cannot store secrets. If for some reason you cannot register a public OAuth2 client, you can use the API to create a token request on your behalf using the client secret. It is recommended to use a public client instead of using this endpoint. POST /auth/oauth2-token is used to exchange an authorization code with an access token.
• GET /auth/ldap-providers is used to list LDAP identity providers. This endpoint requires an admin role because it also displays secrets. Normal users can list visible OAuth2 and LDAP identity providers with GET /auth/providers.
• POST /auth/ldap-providers is to create an LDAP identity provider. Requires admin role. If password is specified, it is stored encrypted.
• PUT /auth/ldap-providers is to update an LDAP identity provider. Requires admin role.
• POST /auth/ldap-providers/<string:provider_name>/login is used to log in to the given LDAP provider with username and password.
• PUT /users/identity-provider is used to update the identity provider that the user uses for login. Inviters can update their invitees' identity providers. Admins can update the identity provider of everyone.
• PUT /users/inviters-providers/<string:username> is used to update the list of allowed identity providers that an inviter can use to invite. Requires admin or inviter role.
• GET /users/inviters-providers/<string:username> is used to display the list of allowed identity providers that an inviter can use to invite. Requires admin or inviter role. If the listed user has no role, this endpoint returns an empty list.
• GET /users/invitation/<string:token> is used to retrieve the metadata attached to the invitation. The invitation must be unused.

Update GAMS to version 41.5.0.

Update GAMS to version 41.4.0.

Update GAMS to version 41.3.0.

Update GAMS to version 41.2.0.

Update GAMS to version 41.1.0.

Update GAMS to version 40.4.0.

Update GAMS to version 40.3.0.

Bug fixes

• Fix issue with Docker Engine version 20.10.18 that caused all model runs to be corrupted.

Update GAMS to version 40.2.0.

Update GAMS to version 40.1.1.

Update GAMS to version 40.1.0.

New features

• Improved logging for worker subprocesses.
• The /cleanup endpoint can now be accessed by all users. Inviters see the job/Hypercube job results from themselves and from all invitees. Normal users see only their own results.

Bug fixes

• Fix issue where worker crashed if the GAMS output contained non-UTF8 characters.
• UI: Fix issue where the job status on the job info page was incorrectly displayed as "Running" even though the job was in "Canceling" status.
• UI: Fix issue where filtering a table did not reset the pagination module.

Update GAMS to version 39.3.0.

New features

• UI: When a job is canceled on the job page, the user is now prompted to confirm the cancellation.
• UI: When deleting a user/invitation, the username/invitation code is now mentioned in the confirmation dialog.
• UI: Timing info is now also available on the job info page for Hypercube jobs.

Bug fixes

• UI: Fix issue where the instance multiplier in usage charts was not considered.

API Changes

• The maximum length for Hypercube job ids is now 255 characters. In addition, ids are now validated against the regular expression ^[0-9a-zA-Z_\-. ]+$. For more information, check the chapter on Hypercube jobs.

Update GAMS to version 39.2.1.

Update GAMS to version 39.2.0.

Update GAMS to version 39.1.1.

Update GAMS to version 39.1.0.

Bug fixes

• Fix issue issue where the event manager could hang indefinitely when receiving a canceled job.

New features

• UI: Add charts when viewing a user's usage data.
• UI: Engine SaaS users can now select a default instance.
• UI: Minor restructuring of job submission form.

Bug fixes

• Fixed an issue in the GET /usage endpoint when recursively querying usage data for an inviter as root admin. In this case, the root admin usage data was also included.

API Changes

• The maximum length for text entries is now 10 MB. It can be configured via the new /configuration endpoint.

New features

• It is now possible to tag a (Hypercube) job with a human-readable string to make it easier to identify.
• Engine licences are now tied to a specific Engine installation. Old licences will continue to be accepted and work just as before. To learn more about how to activate new Engine licenses, read here.
• Engine now supports webhooks as a way to be notified when a job is finished. Webhooks are disabled by default, but can be enabled via the new configuration API (either for administrators only or for everyone).
• When registering models (POST/PATCH /namespaces/{namespace}/models/{model}), you can now specify that model files should be protected from being overwritten (protect_model_files parameter). Together with INEX files, you can thus protect the intellectual property of your models.
• Jobs can now be shared with user groups. Any user with access to this group can then access the job (view the job details, cancel the job, download/delete the results, etc.).

Bug fixes

• The query for usage data (GET /usage/) by the first admin with recursive=true did not contain any usage data of this admin.

Update GAMS to version 38.3.0.

Update GAMS to version 38.2.1.

Update GAMS to version 38.2.0.

Bug Fixes

• Fix issue with sorting by username, model and namespace when listing jobs, listing hypercubes and listing cleanup entries.
• Fix layout issue with Engine UI when many namespaces are present.

Update GAMS to version 38.1.0.

API Changes

• The minimum length for passwords has been increased to 8 characters.
• The maximum length for namespace names has been increased to 100 characters.

Bug fixes

• Fix issue where deleting users did not work in certain situations.
• Fix issue where retrieving user details of a specific user did not work in certain situations.

New features

• The installation script now supports specifying the default admin password via the -a switch.
• UI: String columns in tables rendered client-side (users, models) can now be filtered.

Update GAMS to version 37.1.0.

Breaking Changes

• This release does not contain any breaking changes. In the next 3rd release stream entries and partial logs will return 410 instead of 308 when the stream is ended.

New features

• The installation script now supports specifying the path where GAMS Engine is mounted via the -m switch.
• The access scope of JWT can now be restricted.
• The maximum expiration time of JWT was increased to 186 days.
• JWT can now be invalidated via the new /auth/logout endpoint.
• The worker container now marks the GAMS process as the first candidate for the Out-of-Memory killer.
• The following containers accept the new GMS_RUNNER_LOG_SHOW_MSEC environment variable to add milliseconds to timestamps in log entries:
  • Broker*
  • Worker
  • Cleaner
  • Hypercube Unpacker
  • Hypercube Appender
  • Dependency Checker
  • Job Watcher
  • Job Spawner
  • Job Cleaner
  • Job Canceler
  Setting GMS_RUNNER_LOG_SHOW_MSEC environment variable to 'true' will change the logging format. Not setting it or setting it to anything else will have no effect. For example, a log from the worker container: [17/10/2021 14:58:47] INFO #: Start consuming on signal queue Would become: [17/10/2021 14:58:47.084] INFO #: Start consuming on signal queue

API Changes

• DELETE /jobs/{token}/stream-entry/{entry_name} now returns queue_finished which indicates whether the stream entry queue has been finished.
• GET /hypercube/ labels field also includes resource_warning which can be "none", "memory" or "disk".
• GET /jobs/{token} labels field also includes resource_warning which can be "none", "memory" or "disk".
• GET /usage/ labels field also includes resource_warning which can be "none", "memory" or "disk".
• POST /auth/, creating JWT token via basic auth, is not considered deprecated anymore since it might be used to create a new token using another token.
• GET /users/ now returns the details of the logged in user if user is not admin or inviter. Previously a 403 Forbidden error was thrown if the username parameter was not explicitly set to the name of the logged in user.
• Now it is possible to use stream/text entries to track files located in subdirectories.
  • New endpoint: GET /jobs/{token}/text-entry?entry_name={entry_name}. It does exactly what GET /jobs/{token}/text-entry/{entry_name} does but accepts the entry name in query string to allow querying inside the directories. Operation ID is: queryJobTextEntry
  • New endpoint: HEAD /jobs/{token}/text-entry?entry_name={entry_name}. It does exactly what HEAD /jobs/{token}/text-entry/{entry_name} does but accepts the entry name in query string to allow querying inside the directories. Operation ID is: queryJobTextEntryInfo
  • New endpoint: DELETE /jobs/{token}/stream-entry?entry_name={entry_name}. It does exactly what DELETE /jobs/{token}/stream-entry/{entry_name} does but accepts the entry name in query string to allow querying inside the directories. Operation ID is consumeStreamEntry

Bug fixes

• Fix issue where sparse output of stream entries caused connection timeouts.
• Introduced performance improvements for stream entries.
• CRITICAL: Fix security issue that allowed an authorized attacker to leak system information.

Breaking changes

• The log file is no longer automatically appended to the array of text entries. GAMS log files can get quite large and storing them can consume significant resources. Therefore, this is now opt-in: To save log files, you must explicitly add them to the text_entries array.

New features

• Models can now be assigned to user groups to limit the visibility of the model.
• New string interface for INEX files.
• The results of GET /cleanup/results can now be filtered by namespace.
• Text entries and stream entries can now be specified when creating/updating a model.

Bug fixes

• CRITICAL: Fix a security issue that can lead to privilege escalation if an attacker has access to an inviter account.
• Fix issue with updating models via Python client.

API Changes

• POST /jobs/ accepts query parameter inex_string
that can be used instead of inex_file
• POST /hypercube/ accepts query parameter inex_string that can be used instead of inex_file
• POST /namespaces/{namespace}/models/{model} accepts query parameter inex_string that can be used instead of inex_file
• PATCH /namespaces/{namespace}/models/{model} accepts form parameter inex_string that can be used instead of inex_file
• Admins can no longer make a user an admin if their inviter is not also an admin.

Update GAMS to version 36.2.0.

Bug fixes

• Fix issue with updating models via Engine UI.

New features

• When inviting users, Engine now supports assigning a GAMS license when invited by an admin.
• Now, admins can change usernames. Changing username invalidates previously generated JWT tokens.
• Now, changing password invalidates previously generated JWT tokens.
• Significantly reduced image size of Engine components.

Bug fixes

• Fix issue with inheriting licenses from admins.
• Fix issue where license inheritance did not work in certain situations.

API Changes

• When listing invitations, quotas, user groups, and GAMS license attached to the invitation are also listed.
• All the endpoints that return datetimes ensure that the datetimes have timezone info. For example, if an old return value were 2021-08-04T17:10:15.000000, which did not indicate the timezone, the new return value would be 2021-08-04T17:10:15.000000+00:00.