Procedural Constraints Action/Database Support #1596

skovati · 2024-11-08T17:48:56Z

Tickets addressed: Closes DB updates for Procedural Constraints #1572, Closes Reordering Multiple Invocations of Scheduling Goals Fails #1633
Review: By commit
Merge strategy: Merge (no squash)

Description

Mirroring the database changes that were made for procedural scheduling support, this PR makes the following changes to the constraints sub-schema in merlin:

an auto-generated ID invocation_id was added to constraint_specification and made the new (sole) primary key. This allows for multiple "invocations" of that same constraint definition in the same spec.
type enum, uploaded_jar_id, and parameter schema fields were added to constraint_definition to support procedural constraints
definition was made nullable in constraint_definition, since it will be null when that constraint's source code is contained in the uploaded jar.
constraint_run now holds an additional field, constraint_invocation_id, which replaced (constraint_id, constraint_revision) in the primary key definition. This is so we can have a constraint run for each invocation in a plan spec, which might share constraint definitions and revisions.
constraint_run also holds the "as run" arguments that were passed to the constraint during execution in a new json field arguments.
merlin server was updated to comply with these schema changes
e2e tests were updated to comply as well

Verification

WIP

Documentation

Future work

skovati · 2024-11-08T17:51:24Z

Active works in progress:

database migrations
comments for new tables
verification of migrations against real data

Mythicaeda · 2025-01-03T22:37:55Z

deployment/postgres-init-db/sql/tables/merlin/constraints/constraint_run.sql

@@ -10,7 +12,7 @@ create table merlin.constraint_run (
  requested_at timestamptz not null default now(),

  constraint constraint_run_key
-    primary key (constraint_id, constraint_revision, simulation_dataset_id),
+    primary key (constraint_invocation_id, simulation_dataset_id),


Blocking: This pkey is not sufficient. If the user decides to check constraints twice, updating the definition of a constraint between checks, they will get a pkey conflict when the action attempts to cache the output of the second definition.

(This comment is here to prevent me from not fixing this)

- named `priority` in the DB to maintain schema parity with scheduling, and because `order` is a reserved keyword in SQL - fixes #1633

- add a column to track errors that occur when a constraint is run - replace 'constraint_run' with an association table between 'constraint_request' and 'constraint_results' - update Hasura metadata

Code Pattern now matches the one used for Procedural Constraints, with the definition stored within a Type interface Similarly, Constraint and ConstraintRecord have been merged, as they contain the exact same fields.

Include constraint invocation id in violations response

Now gets the type of the constraint, and the JAR path, in the case of procedural constraint. Changes the ConstraintRecord creation code to use column names to get values rather than depending on the column order of the SQL statement.

Class now has two types: A SuccessType and a FailureType. This allows us to know what type the value is when working with an instance. `getOrNull` has been replaced with `get` Single Constructor has been split into a Success Constructor and a Failure Constructor

The code will throw if the Optional is not present, so the wrapping is unnecessary

…ources Three implementations: 1) EDSLConstraintResult: Formerly ConstraintResult, implementation unchanged aside from adding a toJSON method. 2) DBConstraintResult: A record storing a Constraint Result that originated from the database 3) ProceduralConstraintResult: A record storing a

Changes the output of the plan constraints from a map to a list, as the keyset is only accessed for a `remove` command. Remove ConstraintRunRecord, as DBConstraintResult replaces it

…s Exception>> Refactors ConstraintsCompilationError to be an Extension If a running constraint throws an exception, that exception is now caught and put in the results map Generalizes serializeConstraintCompilationErrors to handle all Constraint Errors

sonarqubecloud · 2025-03-13T15:03:59Z

Quality Gate failed

Failed conditions
4.5% Duplication on New Code (required ≤ 3%)
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

skovati requested a review from Mythicaeda November 8, 2024 17:48

skovati self-assigned this Nov 8, 2024

skovati temporarily deployed to e2e-test November 8, 2024 17:49 — with GitHub Actions Inactive

skovati had a problem deploying to e2e-test November 8, 2024 17:49 — with GitHub Actions Failure

skovati temporarily deployed to e2e-test November 8, 2024 17:49 — with GitHub Actions Inactive

skovati removed the request for review from Mythicaeda November 8, 2024 17:51

Mythicaeda reviewed Jan 3, 2025

View reviewed changes

Mythicaeda force-pushed the feature/procedural-constraints-db-support branch from 141cec3 to 9b55bd7 Compare February 12, 2025 21:11

Mythicaeda had a problem deploying to e2e-test February 12, 2025 21:11 — with GitHub Actions Failure

Mythicaeda force-pushed the feature/procedural-constraints-db-support branch from 9b55bd7 to 615767e Compare February 12, 2025 21:18

Mythicaeda had a problem deploying to e2e-test February 12, 2025 21:18 — with GitHub Actions Failure

Mythicaeda force-pushed the feature/procedural-constraints-db-support branch from 615767e to 770a4c8 Compare February 19, 2025 23:58

Mythicaeda had a problem deploying to e2e-test February 19, 2025 23:59 — with GitHub Actions Failure

Mythicaeda had a problem deploying to e2e-test February 20, 2025 00:02 — with GitHub Actions Failure

Mythicaeda had a problem deploying to e2e-test February 24, 2025 19:13 — with GitHub Actions Failure

Mythicaeda force-pushed the feature/procedural-constraints-db-support branch from 563a1d4 to 4e556a3 Compare February 24, 2025 19:36

Mythicaeda had a problem deploying to e2e-test February 24, 2025 19:36 — with GitHub Actions Failure

Mythicaeda added 21 commits March 13, 2025 07:59

DB: Add ordering support for Constraints

70e221b

- named `priority` in the DB to maintain schema parity with scheduling, and because `order` is a reserved keyword in SQL - fixes #1633

DB: Add tables to track constraint requests. (BREAKING API CHANGE)

3193c44

- add a column to track errors that occur when a constraint is run - replace 'constraint_run' with an association table between 'constraint_request' and 'constraint_results' - update Hasura metadata

DB: Database Migration

66cbed2

Cleanup whitespace in Constraint.kt

f4e699b

Extract parseJson into MerlinParsers

c938996

Update ConstraintRecord

d1cf128

Code Pattern now matches the one used for Procedural Constraints, with the definition stored within a Type interface Similarly, Constraint and ConstraintRecord have been merged, as they contain the exact same fields.

Merge ConstraintRecord and Constraint types cleanup

5575903

Include constraint invocation id in violations response

Update GetPlanConstraintsAction to support Types

54a1f55

Now gets the type of the constraint, and the JAR path, in the case of procedural constraint. Changes the ConstraintRecord creation code to use column names to get values rather than depending on the column order of the SQL statement.

Add optional "force" flag to Check Constraints endpoint

90e27ea

Refactor Fallible

03968b5

Class now has two types: A SuccessType and a FailureType. This allows us to know what type the value is when working with an instance. `getOrNull` has been replaced with `get` Single Constructor has been split into a Success Constructor and a Failure Constructor

ConstraintAction remove Optional wrapping on SimulationResultsHandle

108c6d8

The code will throw if the Optional is not present, so the wrapping is unnecessary

Add method to convert Procedural Interval to EDSL Interval

be7ae89

Compile and Run Procedural Constraints

60a7695

Update build.gradles

94900e5

Extract Parameter Schemas for Procedural Constraints

e1dfedf

Store the outcome of a Constraint Run in the DB

cb69e0f

Load cached constraint results

e36a90e

Changes the output of the plan constraints from a map to a list, as the keyset is only accessed for a `remove` command. Remove ConstraintRunRecord, as DBConstraintResult replaces it

Remove Unused Serializers in ResponseSerializers

446c075

initial work to update e2e tests with constraint invocation ids

4b76798

Mythicaeda force-pushed the feature/procedural-constraints-db-support branch from ff2b995 to 4b76798 Compare March 13, 2025 15:02

Mythicaeda had a problem deploying to e2e-test March 13, 2025 15:02 — with GitHub Actions Failure

Mythicaeda temporarily deployed to e2e-test March 13, 2025 15:02 — with GitHub Actions Inactive

Mythicaeda had a problem deploying to e2e-test March 13, 2025 15:57 — with GitHub Actions Failure

Mythicaeda changed the title ~~Procedural Constraints Database Support~~ Procedural Constraints Action/Database Support Mar 13, 2025

Mythicaeda had a problem deploying to e2e-test March 13, 2025 15:57 — with GitHub Actions Failure

Mythicaeda had a problem deploying to e2e-test March 13, 2025 15:58 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Procedural Constraints Action/Database Support #1596

Procedural Constraints Action/Database Support #1596

skovati commented Nov 8, 2024 •

edited by Mythicaeda

Loading

skovati commented Nov 8, 2024 •

edited

Loading

Mythicaeda Jan 3, 2025 •

edited

Loading

sonarqubecloud bot commented Mar 13, 2025

Procedural Constraints Action/Database Support #1596

Are you sure you want to change the base?

Procedural Constraints Action/Database Support #1596

Conversation

skovati commented Nov 8, 2024 • edited by Mythicaeda Loading

Description

Verification

Documentation

Future work

skovati commented Nov 8, 2024 • edited Loading

Mythicaeda Jan 3, 2025 • edited Loading

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 13, 2025

Quality Gate failed

skovati commented Nov 8, 2024 •

edited by Mythicaeda

Loading

skovati commented Nov 8, 2024 •

edited

Loading

Mythicaeda Jan 3, 2025 •

edited

Loading