Simple support for remote web stores

doc/sphinx-guides/source/installation/config.rst

@@ -238,13 +238,15 @@ As for the "Remote only" authentication mode, it means that:
 - ``:DefaultAuthProvider`` has been set to use the desired authentication provider
 - The "builtin" authentication provider has been disabled (:ref:`api-toggle-auth-provider`). Note that disabling the "builtin" authentication provider means that the API endpoint for converting an account from a remote auth provider will not work. Converting directly from one remote authentication provider to another (i.e. from GitHub to Google) is not supported. Conversion from remote is always to "builtin". Then the user initiates a conversion from "builtin" to remote. Note that longer term, the plan is to permit multiple login options to the same Dataverse installation account per https://github.com/IQSS/dataverse/issues/3487 (so all this talk of conversion will be moot) but for now users can only use a single login option, as explained in the :doc:`/user/account` section of the User Guide. In short, "remote only" might work for you if you only plan to use a single remote authentication provider such that no conversion between remote authentication providers will be necessary.
 
-File Storage: Using a Local Filesystem and/or Swift and/or object stores
-------------------------------------------------------------------------
+File Storage: Using a Local Filesystem and/or Swift and/or object stores and/or trusted remote services
+-------------------------------------------------------------------------------------------------------
 
 By default, a Dataverse installation stores all data files (files uploaded by end users) on the filesystem at ``/usr/local/payara5/glassfish/domains/domain1/files``. This path can vary based on answers you gave to the installer (see the :ref:`dataverse-installer` section of the Installation Guide) or afterward by reconfiguring the ``dataverse.files.\<id\>.directory`` JVM option described below.
 
 A Dataverse installation can alternately store files in a Swift or S3-compatible object store, and can now be configured to support multiple stores at once. With a multi-store configuration, the location for new files can be controlled on a per-Dataverse collection basis.
 
+Dataverse may also be configured to reference some files (e.g. large and/or sensitive data) stored in a trusted remote web-accessible system.
+
 The following sections describe how to set up various types of stores and how to configure for multiple stores.
 
 Multi-store Basics
@@ -663,6 +665,34 @@ Migrating from Local Storage to S3
 
 Is currently documented on the :doc:`/developers/deployment` page.
 
+Trusted Remote Storage
+++++++++++++++++++++++
+
+In addition to having the type "remote" and requiring a label, Trusted Remote Stores are defined in terms of a baseURL - all files managed by this store must be at a path starting with this URL, and a baseStore - a file, s3, or swift store that can be used to store additional ancillary dataset files (e.g. metadata exports, thumbnails, auxiliary files, etc.).
+These and other available options are described in the table below.
+
+Remote stores can range from being a static trusted website to a sophisticated service managing access requests and logging activity
+and/or managing access to a secure enclave. For specific remote stores, consult their documentation when configuring the remote store in Dataverse.
+
+.. table::
+    :align: left
+
+    ===========================================  ==================  ==========================================================================  =============
+    JVM Option                                   Value               Description                                                                 Default value
+    ===========================================  ==================  ==========================================================================  =============
+    dataverse.files.<id>.type                    ``remote``          **Required** to mark this storage as remote.                                (none)
+    dataverse.files.<id>.label                   <?>                 **Required** label to be shown in the UI for this storage                   (none)
+    dataverse.files.<id>.baseUrl                 <?>                 **Required** All files must have URLs of the form <baseUrl>/*               (none)
+    dataverse.files.<id>.baseStore               <?>                 **Required** The id of a base store (of type file, s3, or swift)            (none)
+    dataverse.files.<id>.download-redirect       ``true``/``false``  Enable direct download (should usually be true).                            ``false``
+    dataverse.files.<id>.secretKey               <?>                 A key used to sign download requests sent to the remote store. Optional.    (none)
+    dataverse.files.<id>.url-expiration-minutes  <?>                 If direct downloads and using signing: time until links expire. Optional.   60
+    dataverse.files.<id>.remote-store-name       <?>                 A short name used in the UI to indicate where a file is located. Optional   (none)
+    dataverse.files.<id>.remote-store-url        <?>                 A url to an info page about the remote store used in the UI. Optional.      (none)
+
+    ===========================================  ==================  ==========================================================================  =============
+
+
 
 .. _Branding Your Installation:
 
@@ -2568,7 +2598,7 @@ Number of errors to display to the user when creating DataFiles from a file uplo
 .. _:BagItHandlerEnabled:
 
 :BagItHandlerEnabled
-+++++++++++++++++++++
+++++++++++++++++++++
 
 Part of the database settings to configure the BagIt file handler. Enables the BagIt file handler. By default, the handler is disabled.
 

src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java

@@ -15,7 +15,6 @@
 public abstract class DvObjectContainer extends DvObject {
 
 
-    //Default to "file" is for tests only
     public static final String UNDEFINED_METADATA_LANGUAGE_CODE = "undefined"; //Used in dataverse.xhtml as a non-null selection option value (indicating inheriting the default)
 
 

src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java

@@ -1933,7 +1933,7 @@ private void handleReplaceFileUpload(String fullStorageLocation,
 
         fileReplacePageHelper.resetReplaceFileHelper();
         saveEnabled = false;
-        String storageIdentifier = DataAccess.getStorarageIdFromLocation(fullStorageLocation);
+        String storageIdentifier = DataAccess.getStorageIdFromLocation(fullStorageLocation);
         if (fileReplacePageHelper.handleNativeFileUpload(null, storageIdentifier, fileName, contentType, checkSumValue, checkSumType)) {
             saveEnabled = true;
 
@@ -2078,8 +2078,12 @@ public void handleExternalUpload() {
         if (!checksumTypeString.isBlank()) {
             checksumType = ChecksumType.fromString(checksumTypeString);
         }
+
+        //Should only be one colon with curent design
         int lastColon = fullStorageIdentifier.lastIndexOf(':');
-        String storageLocation = fullStorageIdentifier.substring(0, lastColon) + "/" + dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage() + "/" + fullStorageIdentifier.substring(lastColon + 1);
+        String storageLocation = fullStorageIdentifier.substring(0,lastColon) + "/" + dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage() + "/" + fullStorageIdentifier.substring(lastColon+1);
+        storageLocation = DataAccess.expandStorageIdentifierIfNeeded(storageLocation);
+
         if (uploadInProgress.isFalse()) {
             uploadInProgress.setValue(true);
         }
@@ -3044,7 +3048,7 @@ public void saveAdvancedOptions() {
     }
 
     public boolean rsyncUploadSupported() {
-        // ToDo - rsync was written before multiple store support and currently is hardcoded to use the "s3" store. 
+        // ToDo - rsync was written before multiple store support and currently is hardcoded to use the DataAccess.S3 store. 
         // When those restrictions are lifted/rsync can be configured per store, the test in the 
         // Dataset Util method should be updated
         if (settingsWrapper.isRsyncUpload() && !DatasetUtil.isAppropriateStorageDriver(dataset)) {

src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java

@@ -560,12 +560,12 @@ public void addFileToCustomZipJob(String key, DataFile dataFile, Timestamp times
 
     public String getDirectStorageLocatrion(String storageLocation) {
         String storageDriverId;
-        int separatorIndex = storageLocation.indexOf("://");
+        int separatorIndex = storageLocation.indexOf(DataAccess.SEPARATOR);
         if ( separatorIndex > 0 ) {
             storageDriverId = storageLocation.substring(0,separatorIndex);
 
             String storageType = DataAccess.getDriverType(storageDriverId);
-            if ("file".equals(storageType) || "s3".equals(storageType)) {
+            if (DataAccess.FILE.equals(storageType) || DataAccess.S3.equals(storageType)) {
                 return storageType.concat(storageLocation.substring(separatorIndex));
             }
         }

src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

@@ -1908,7 +1908,7 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String
                         String message = wr.getMessage();
                         return error(Response.Status.INTERNAL_SERVER_ERROR, "Uploaded files have passed checksum validation but something went wrong while attempting to put the files into Dataverse. Message was '" + message + "'.");
                     }
-                } else if(storageDriverType.equals("s3")) {
+                } else if(storageDriverType.equals(DataAccess.S3)) {
 
                     logger.log(Level.INFO, "S3 storage driver used for DCM (dataset id={0})", dataset.getId());
                     try {
@@ -2371,40 +2371,40 @@ public Response addFileToDataset(@PathParam("id") String idSupplied,
         String newFilename = null;
         String newFileContentType = null;
         String newStorageIdentifier = null;
-		if (null == contentDispositionHeader) {
-			if (optionalFileParams.hasStorageIdentifier()) {
-				newStorageIdentifier = optionalFileParams.getStorageIdentifier();
-				// ToDo - check that storageIdentifier is valid
-				if (optionalFileParams.hasFileName()) {
-					newFilename = optionalFileParams.getFileName();
-					if (optionalFileParams.hasMimetype()) {
-						newFileContentType = optionalFileParams.getMimeType();
-					}
-				}
-			} else {
-				return error(BAD_REQUEST,
-						"You must upload a file or provide a storageidentifier, filename, and mimetype.");
-			}
-		} else {
-			newFilename = contentDispositionHeader.getFileName();
-                        // Let's see if the form data part has the mime (content) type specified. 
-                        // Note that we don't want to rely on formDataBodyPart.getMediaType() - 
-                        // because that defaults to "text/plain" when no "Content-Type:" header is 
-                        // present. Instead we'll go through the headers, and see if "Content-Type:" 
-                        // is there. If not, we'll default to "application/octet-stream" - the generic
-                        // unknown type. This will prompt the application to run type detection and 
-                        // potentially find something more accurate.
-                        //newFileContentType = formDataBodyPart.getMediaType().toString();
-
-                        for (String header : formDataBodyPart.getHeaders().keySet()) {
-                            if (header.equalsIgnoreCase("Content-Type")) {
-                                newFileContentType = formDataBodyPart.getHeaders().get(header).get(0);
-                            }
-                        }
-                        if (newFileContentType == null) {
-                            newFileContentType = FileUtil.MIME_TYPE_UNDETERMINED_DEFAULT;
-                        }
-		}
+        if (null == contentDispositionHeader) {
+            if (optionalFileParams.hasStorageIdentifier()) {
+                newStorageIdentifier = optionalFileParams.getStorageIdentifier();
+                newStorageIdentifier = DataAccess.expandStorageIdentifierIfNeeded(newStorageIdentifier);
+                if (optionalFileParams.hasFileName()) {
+                    newFilename = optionalFileParams.getFileName();
+                    if (optionalFileParams.hasMimetype()) {
+                        newFileContentType = optionalFileParams.getMimeType();
+                    }
+                }
+            } else {
+                return error(BAD_REQUEST,
+                        "You must upload a file or provide a storageidentifier, filename, and mimetype.");
+            }
+        } else {
+            newFilename = contentDispositionHeader.getFileName();
+            // Let's see if the form data part has the mime (content) type specified.
+            // Note that we don't want to rely on formDataBodyPart.getMediaType() -
+            // because that defaults to "text/plain" when no "Content-Type:" header is
+            // present. Instead we'll go through the headers, and see if "Content-Type:"
+            // is there. If not, we'll default to "application/octet-stream" - the generic
+            // unknown type. This will prompt the application to run type detection and
+            // potentially find something more accurate.
+            // newFileContentType = formDataBodyPart.getMediaType().toString();
+
+            for (String header : formDataBodyPart.getHeaders().keySet()) {
+                if (header.equalsIgnoreCase("Content-Type")) {
+                    newFileContentType = formDataBodyPart.getHeaders().get(header).get(0);
+                }
+            }
+            if (newFileContentType == null) {
+                newFileContentType = FileUtil.MIME_TYPE_UNDETERMINED_DEFAULT;
+            }
+        }
 
 
         //-------------------
@@ -2912,7 +2912,7 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf,
         }
         if (!user.isSuperuser()) {
             return error(Response.Status.FORBIDDEN, "Superusers only.");
-    	}
+        }
 
         Dataset dataset; 
 
@@ -2930,7 +2930,7 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf,
                 return ok("Storage driver set to: " + store.getKey() + "/" + store.getValue());
             }
         }
-    	return error(Response.Status.BAD_REQUEST,
+        return error(Response.Status.BAD_REQUEST,
             "No Storage Driver found for : " + storageDriverLabel);
     }
 
@@ -2948,7 +2948,7 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf,
         }
         if (!user.isSuperuser()) {
             return error(Response.Status.FORBIDDEN, "Superusers only.");
-    	}
+        }
 
         Dataset dataset; 
 
@@ -2960,7 +2960,7 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf,
 
         dataset.setStorageDriverId(null);
         datasetService.merge(dataset);
-    	return ok("Storage reset to default: " + DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER);
+        return ok("Storage reset to default: " + DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER);
     }
 
     @GET

src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java

@@ -93,7 +93,7 @@ public void writeTo(DownloadInstance di, Class<?> clazz, Type type, Annotation[]
 
                 // Before we do anything else, check if this download can be handled 
                 // by a redirect to remote storage (only supported on S3, as of 5.4):
-                if (storageIO instanceof S3AccessIO && ((S3AccessIO) storageIO).downloadRedirectEnabled()) {
+                if (storageIO.downloadRedirectEnabled()) {
 
                     // Even if the above is true, there are a few cases where a  
                     // redirect is not applicable. 
@@ -188,16 +188,15 @@ public void writeTo(DownloadInstance di, Class<?> clazz, Type type, Annotation[]
                         // [attempt to] redirect: 
                         String redirect_url_str;
                         try {
-                            redirect_url_str = ((S3AccessIO) storageIO).generateTemporaryS3Url(auxiliaryTag, auxiliaryType, auxiliaryFileName);
+                            redirect_url_str = storageIO.generateTemporaryDownloadUrl(auxiliaryTag, auxiliaryType, auxiliaryFileName);
                         } catch (IOException ioex) {
+                            logger.warning("Unable to generate downloadURL for " + dataFile.getId() + ": " + auxiliaryTag);
+                            //Setting null will let us try to get the file/aux file w/o redirecting 
                             redirect_url_str = null;
                         }
 
-                        if (redirect_url_str == null) {
-                            throw new ServiceUnavailableException();
-                        }
-
                         logger.fine("Data Access API: direct S3 url: " + redirect_url_str);
+
                         URI redirect_uri;
 
                         try {

src/main/java/edu/harvard/iq/dataverse/api/util/JsonResponseBuilder.java

@@ -222,6 +222,7 @@ public JsonResponseBuilder log(Logger logger, Level level, Optional<Throwable> e
         metadata.deleteCharAt(metadata.length()-1);
 
         if (ex.isPresent()) {
+            ex.get().printStackTrace();
             metadata.append("|");
             logger.log(level, metadata.toString(), ex);
             if(includeStackTrace) {