Feat/parameter scan #74
Conversation
There was a problem hiding this comment.
This is a good first stab at creating a custom parameter scan function 👏.
However, there are a few pieces that are missing. The biggest is the ability to explore what species are most sensitive to the parameter changes, which currently cannot be done since only a single species saved. I get that this simplifies the visualization and data storage needs, but also limits the usefulness of the tool.
The basico documentation https://copasi.org/Support/User_Manual/Tasks/Parameter_Scan/ provides a good overview of their widget strategy. However, I think we can do much better than this by providing an interactive environment for users to explore the results. Please see if my suggestion for making use of the multiple simulation feature and plotting across different simulations is possible. If not, let's discuss some alternatives....
| # check if the param_name is not None | ||
| if param_name is None: | ||
| continue | ||
| # if param is a kinectic parameter |
There was a problem hiding this comment.
| # if param is a kinectic parameter | |
| # if param is a kinetic parameter |
| description="species concentration at the time point") | ||
|
|
||
| @dataclass | ||
| class RecurringData: |
There was a problem hiding this comment.
| class RecurringData: | |
| class ReoccurringData: |
| class RecurringData: | ||
| """ | ||
| Dataclass for storing the species and time data | ||
| on recurring basis. |
There was a problem hiding this comment.
| on recurring basis. | |
| on a reoccurring basis. |
| species_data: SpeciesData = Field( | ||
| description="species name and initial concentration data", | ||
| default=None) | ||
| recurring_data: RecurringData = Field( |
There was a problem hiding this comment.
| recurring_data: RecurringData = Field( | |
| reoccurring_data: ReoccurringData = Field( |
| species_data: SpeciesData = Field( | ||
| description="species name and initial concentration data", | ||
| default=None) | ||
| recurring_data: RecurringData = Field( |
There was a problem hiding this comment.
| recurring_data: RecurringData = Field( | |
| reocurring_data: ReoccurringData = Field( |
|
|
||
| def add_rec_events(model_object, recurring_data): | ||
| """ | ||
| Add reocurring events to the model. |
There was a problem hiding this comment.
| Add reocurring events to the model. | |
| Add reoccurring events to the model. |
| """ | ||
| Dataclass for storing the parameter scan data. | ||
| """ | ||
| species_name: str = Field(description="Species name to investigate", |
There was a problem hiding this comment.
Is there a particular reason to limit the saved data to a single species? In principle, if we are running the simulations for each and every parameter step, we should have access to all of the species concentrations over time, correct?
I would recommend saving the simulation results for each and every parameter scan step. This could be achieved using the latest feature for multiple simulations where the name of the simulation is the parameter scan step (which I see you used below to define the column in the dataframe). The challenge will then be visualizing the results for the user...
I think the most straight forward way would be for the user to prompt the model to create a custom plot based on what they specifically would like to see. For example, after requesting a parameter scan:
assistant: The parameter scan has finished running. There is a lot of data to visualize! Please specify what species and from which parameter scan steps you would like me to plot. For example, if you ask "plot species X and Y across all parameter scans", I will create a time vs concentration plot for species X and Y with additional labels on X and Y to indicate the parameter step.
There was a problem hiding this comment.
@dmccloskey please correct me if I am wrong. Do you mean to run a parameter scan of more than one species in the same prompt? If that is the case, then it is already possible. Sorry, I forgot to add it to the demo earlier. Please take a look at the video below where I ask the agent to run a param scan (kIL6RBind) on Ab{serum} and IL6{serum}. In this case, the agent invokes the tool twice once with each species. And, as you said above, it generates a name for each time the tool is invoked and saves them in the state's key. Users can do it by specifying multiple species.
T2Bdemo6.mp4
There was a problem hiding this comment.
But this means it needs to re-run the same simulation twice then?
There was a problem hiding this comment.
@dmccloskey
Yes, for now... I'll rework the backend. But is the front-end rendering as expected?
There was a problem hiding this comment.
Based on the code, it is rendering as I would expect. However, as I mentioned, I believe we are short changing ourselves on providing the type of interactive experience that is lacking from tools like Copasi and Basico. The ability to generate plots on the fly to address a specific user question is a huge selling point of our platform. Perhaps this will come with the plotting agent.
| @@ -119,6 +119,68 @@ def test_simulate_model_tool(): | |||
| # Check if the data of the second model contains | |||
| assert 'mTORC2' in dic_simulated_data[1]['data'] | |||
|
|
|||
| def test_param_scane_tool(): | |||
| ''' | |||
There was a problem hiding this comment.
The explanation is much appreciated 😊
| @@ -35,18 +35,21 @@ def get_model_metadata(self) -> Dict[str, Union[str, int]]: | |||
| Returns: | |||
| dict: Dictionary with model metadata | |||
| """ | |||
| @abstractmethod | |||
| def update_parameters(self, parameters: Dict[str, Union[float, int]]) -> None: | |||
There was a problem hiding this comment.
I like the choice to split these two methods 👍
|
@dmccloskey Based on our discussions, I have updated the tool to consider all the species names requested by the user in a single invocation. |
|
🎉 This PR is included in version 1.11.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
For authors
Description
T2Bdemo4.mp4
This PR introduces a new
parameter_scantool. Please take a look at the demo video above. When an incorrect parameter name is provided (e.g.,kIL6RBINDinstead ofkIL6RBind), the agent invokes theget_modelinfotool to extract the correct parameter name and automatically reinvokesparameter_scan. This behaviour is demonstrated in the demo and supported by tests. Additionally, when a non-existent species name or parameter is entered (for exampleABinstead ofAb{serum}), the agent notifies the user of the error and suggests valid alternatives from the model. These features enable the agent/tool to handle minor inaccuracies intelligently and provide helpful feedback when corrections are not possible.Fixes # (issue) NA
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests you conducted to verify your changes. These may involve creating new test scripts or updating existing ones.
testsfoldertests/testX.py)Checklist
testsfolder) that prove my fix is effective or that my feature worksFor reviewers
Checklist pre-approval
Checklist post-approval
developintomain? If so, please make sure to add a prefix (feat/fix/chore) and/or a suffix BREAKING CHANGE (if it's a major release) to your commit message.Checklist post-merge
developintomainand is it suppose to run an automated release workflow (if applicable)? If so, please make sure to check under the "Actions" tab to see if the workflow has been initiated, and return later to verify that it has completed successfully.