Remove domain model dependency from the compilation step and add it at deployment time

[scp93ch]

We are currently part way through a process of changing the way domain models are handled. The knowledgebase manager page is able to upload a new (additional) domain model zip bundle including icon files, but at compile time we still have a dependency on a domain model and the previous loading mechanism is used.

• remove the compile-time domain model dependency
• ensure that the dashboard and knowledgebase manager pages work if there is no domain model loaded
• display a warning in the dashboard if no domain model is loaded, along with a link to the knowledgebase manager page
• at first boot time make the system-modeller service look for domain model zip bundles (i.e. *.zip) in a specific filesystem location and load them all in; the filesystem location should be configured in application.properties so that it can be overridden with an environment variable as necessary
• ensure that domain models (including icons, etc) loaded by the new method (whether at boot or runtime) are located in persistent volumes
• remove the old domain model update and download options from the knowledgebase manager
• provide an example bind-mount in the docker-compose file
• document the boot-time domain model import mechanism

[scp93ch]

Regarding the point on "remove the old domain model update and download options from the knowledgebase manager", the current knowledgebase manager page has the following:

The "Download ontologies.json" link needs removing as the list of domain models is now dynamic. The "Download" and "Upload new version" buttons against a specific domain model just deal with the NQ file and should probably be removed as I can't see why someone would want to do that now. If there is a new version of a domain model then it may well have new icons so just downloading/uploading the NQ isn't going to help. I guess there's a possibility that a domain model developer might want to just change the NQ. If @mike1813 thinks they are useful then let's hide those buttons unless you're in the developer mode (and also clarify that they are just about the NQ).

[mike1813]

As far as I can see:

• Delete button: will still be needed if we want the ability to delete a previously deployed domain model.
• Access List button: will still be needed if we want the ability to restrict access to specific domain models. Whether that should be the case is a separate discussion. For now we should keep the button.
• Upload new version: is not needed, as we can use the new mechanism to upload a replacement domain model ZIP bundle.
• Download button: is sometimes useful if there is some uncertainty about which version of a domain model is loaded.

I have used the Download button occasionally to get the (gzipped) deployed NQ file so I can do a text comparison to check if it is the version it claims to be. The reason this was necessary is because the domain model owl#versionInfo property was inserted by hand, so it was possible through human error for two different versions to have the same owl#versionInfo.

In future, that motive for retaining a Download button will go away, as the domain model owl#versionInfo will be controlled by the CI pipeline, so human error will no longer be a factor. The only reasons for keeping this button now are:

• because we are still working with old domain models that might be affected by previous human errors or
• because we want people to be able to download the domain model they are using from system modeller

The first reason to keep the button is not a good one. In principle, we should only be using old models from the archive on the IT Innovation security project internal SharePoint, without alteration. If we need to change any of these models, we should add more 'live development' branches to the GitHub domain model project and tag the updates so the version info is automated. E.g. if we needed an update of the old v5e2-7-39, we would create a dev branch '5e', commit the v5e2-7-39 fileset, and go from there.

The second reason is also not a good one. We should be directing people to get domain model versions from the GitHub domain model project. But this page is restricted to service administrators, so most of those people can't see it.

On that basis, I would remove the 'Upload' and 'Download' button from this page, so we only have the 'Delete' button and (at least for now) the 'Access List' button.

[mike1813]

Instead of the 'Download' button, we could provide a link allowing users (not service administrators) to get a domain model. This should be added to the Dashboard or system model Properties, probably as a link under the domain model version info. This link could either:

• take users directly to the right URL to get the domain model bundle from the relevant repository, or
• open a pop-up displaying a Git command to pull the domain model source files from GitHub

The domain model owl#versionInfo is inserted by csv2nq based on a tag in the domain model GitHub repository, but it isn't just the tag value - some extra strings may be included by csv2nq based on the command line options used. On that basis, it probably makes sense if csv2nq inserts a new property something like 'core#getDomainModel`, removing the need for system modeller to figure that out based on the version info.

I recommend that @scp93ch should review this suggestion and create new issues if this new 'user accessible KB download' option is something we should pursue. This depends on whether csv2nq could know the link or git command needed at the point when a CI pipeline is executed. I don't know enough about the CI pipeline to comment on that.

[kenmeacham]

I've removed the dependency on domain-network artifact. Also gone further to remove ontologies.json and the main asset icons that were in the source tree (except fallback.svg). Functionality to download or update ontologies.json has also been removed.

[kenmeacham]

Regarding 'the following:

ensure that the dashboard and knowledgebase manager pages work if there is no domain model loaded

I've checked that the pages basically work without error. The knowledgebase manager displays nothing except the "Upload new knowledgebase", as you would expect. In the dashboard, you cannot currently create a new model when there is no domain model available (the "Create New Model" button is disabled").

TODO: add some kind of warning that there is no domain model loaded, etc

[kenmeacham]

This isssue is now ready to be reviewed, so I'll submit a pull request shortly..

The new functionality for installing default knowledgebase(s) should be tested. I have made fixes to the README file which hopefully explain the process. The domain-network dependency is now removed, so configuration related to that has also been removed. The ontologies.json file no longer exists, and all support for upload/download of this has been removed, along with the original static images.

Regarding the old domain model update mechanism, I have removed support for uploading a domain model .nq or .rdf file. There was a misconception that update of the knowledgebase did not support .zip bundles, however this has been working for some time.

I have improved the checking of uploaded zip bundles as follows:

1. "Upload new knowledgebase" now checks that the knowledgebase does not already exist, otherwise it throws an error
2. "Upload new version" ensures that the uploaded knowledgebase graph URI matches that requested in the submission