Skip to content

Commit

Permalink
Merge pull request #9204 from QualitativeDataRepository/4959-Allow_in…
Browse files Browse the repository at this point in the history 
…stallations_to_determine_order_of_files_on_Dataset_page

IQSS/4959-Support-By-Category-and-Folder-Grouping
  • Loading branch information
@kcondon
kcondon authored May 2, 2023
2 parents 8181854 + e928ff0 commit bb01984
Show file tree
Hide file tree
Showing 11 changed files with 391 additions and 107 deletions.
9 changes: 9 additions & 0 deletions doc/release-notes/9204-group-by-file-ordering.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
### Support for Grouping Dataset Files by Folder and Category Tag

Dataverse now supports grouping dataset files by folder and/or optionally by Tag/Category. The default for whether to order by folder can be changed via :OrderByFolder. Ordering by category must be enabled by an administrator via the :CategoryOrder parameter which is used to specify which tags appear first (e.g. to put Documentation files before Data or Code files, etc.) These Group-By options work with the existing sort options, i.e. sorting alphabetically means that files within each folder or tag group will be sorted alphabetically. :AllowUsersToManageOrdering can be set to true to allow users to turn folder ordering and category ordering (if enabled) on or off in the current dataset view.

### New Setting

:CategoryOrder - a comma separated list of Category/Tag names defining the order in which files with those tags should be displayed. The setting can include custom tag names along with the pre-defined defaults ( Documentation, Data, and Code, which can be overridden by the ::FileCategories setting.)
:OrderByFolder - defaults to true - whether to group files in the same folder together
:AllowUserManagementOfOrder - default false - allow users to toggle ordering on/off in the dataset display
19 changes: 19 additions & 0 deletions doc/sphinx-guides/source/installation/config.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3710,6 +3710,8 @@ For example:

When set to ``true``, this setting allows a superuser to publish and/or update Dataverse collections and datasets bypassing the external validation checks (specified by the settings above). In an event where an external script is reporting validation failures that appear to be in error, this option gives an admin with superuser privileges a quick way to publish the dataset or update a collection for the user.

.. _:FileCategories:

:FileCategories
+++++++++++++++

Expand Down Expand Up @@ -3811,4 +3813,21 @@ To use the current GDCC version directly:

``curl -X PUT -d 'https://gdcc.github.io/dvwebloader/src/dvwebloader.html' http://localhost:8080/api/admin/settings/:WebloaderUrl``

:CategoryOrder
++++++++++++++

A comma separated list of Category/Tag names defining the order in which files with those tags should be displayed.
The setting can include custom tag names along with the pre-defined tags(Documentation, Data, and Code are the defaults but the :ref:`:FileCategories` setting can be used to use a different set of tags).
The default is category ordering disabled.

:OrderByFolder
++++++++++++++

A true(default)/false option determining whether datafiles listed on the dataset page should be grouped by folder.

:AllowUserManagementOfOrder
+++++++++++++++++++++++++++

A true/false (default) option determining whether the dataset datafile table display includes checkboxes enabling users to turn folder ordering and/or category ordering (if an order is defined by :CategoryOrder) on and off dynamically.

.. _supported MicroProfile Config API source: https://docs.payara.fish/community/docs/Technical%20Documentation/MicroProfile/Config/Overview.html
164 changes: 85 additions & 79 deletions src/main/java/edu/harvard/iq/dataverse/DatasetPage.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
import edu.harvard.iq.dataverse.util.ArchiverUtil;
import edu.harvard.iq.dataverse.util.BundleUtil;
import edu.harvard.iq.dataverse.util.DataFileComparator;
import edu.harvard.iq.dataverse.util.FileSortFieldAndOrder;
import edu.harvard.iq.dataverse.util.FileUtil;
import edu.harvard.iq.dataverse.util.JsfHelper;
Expand Down Expand Up @@ -505,6 +506,16 @@ public void setRemoveUnusedTags(boolean removeUnusedTags) {

private String fileSortField;
private String fileSortOrder;
private boolean tagPresort = true;
private boolean folderPresort = true;
// Due to what may be a bug in PrimeFaces, the call to select a new page of
// files appears to reset the two presort booleans to false. The following
// values are a flag and duplicate booleans to remember what the new values were
// so that they can be set only in real checkbox changes. Further comments where
// these are used.
boolean isPageFlip = false;
private boolean newTagPresort = true;
private boolean newFolderPresort = true;

public List<Entry<String,String>> getCartList() {
if (session.getUser() instanceof AuthenticatedUser) {
Expand Down Expand Up @@ -673,64 +684,43 @@ private List<FileMetadata> selectFileMetadatasForDisplay() {
// We run the search even if no search term and/or facets are
// specified - to generate the facet labels list:
searchResultsIdSet = getFileIdsInVersionFromSolr(workingVersion.getId(), this.fileLabelSearchTerm);
// But, if no search terms were specified, we can immediately return the full
// But, if no search terms were specified, we return the full
// list of the files in the version:
if (StringUtil.isEmpty(fileLabelSearchTerm)
&& StringUtil.isEmpty(fileTypeFacet)
&& StringUtil.isEmpty(fileAccessFacet)
&& StringUtil.isEmpty(fileTagsFacet)) {
if ((StringUtil.isEmpty(fileSortField) || fileSortField.equals("name")) && StringUtil.isEmpty(fileSortOrder)) {
return workingVersion.getFileMetadatasSorted();
} else {
searchResultsIdSet = null;
}
// Since the search results should include the full set of fmds if all the
// terms/facets are empty, setting them to null should just be
// an optimization for the loop below
searchResultsIdSet = null;
}

} else {
// No, this is not an indexed version.
// If the search term was specified, we'll run a search in the db;
// if not - return the full list of files in the version.
// (no facets without solr!)
if (StringUtil.isEmpty(this.fileLabelSearchTerm)) {
if ((StringUtil.isEmpty(fileSortField) || fileSortField.equals("name")) && StringUtil.isEmpty(fileSortOrder)) {
return workingVersion.getFileMetadatasSorted();
}
} else {
if (!StringUtil.isEmpty(this.fileLabelSearchTerm)) {
searchResultsIdSet = getFileIdsInVersionFromDb(workingVersion.getId(), this.fileLabelSearchTerm);
}
}

List<FileMetadata> retList = new ArrayList<>();

for (FileMetadata fileMetadata : workingVersion.getFileMetadatasSorted()) {
for (FileMetadata fileMetadata : workingVersion.getFileMetadatas()) {
if (searchResultsIdSet == null || searchResultsIdSet.contains(fileMetadata.getDataFile().getId())) {
retList.add(fileMetadata);
}
}

if ((StringUtil.isEmpty(fileSortOrder) && !("name".equals(fileSortField)))
|| ("desc".equals(fileSortOrder) || !("name".equals(fileSortField)))) {
sortFileMetadatas(retList);

}

sortFileMetadatas(retList);
return retList;
}

private void sortFileMetadatas(List<FileMetadata> fileList) {
if ("name".equals(fileSortField) && "desc".equals(fileSortOrder)) {
Collections.sort(fileList, compareByLabelZtoA);
} else if ("date".equals(fileSortField)) {
if ("desc".equals(fileSortOrder)) {
Collections.sort(fileList, compareByOldest);
} else {
Collections.sort(fileList, compareByNewest);
}
} else if ("type".equals(fileSortField)) {
Collections.sort(fileList, compareByType);
} else if ("size".equals(fileSortField)) {
Collections.sort(fileList, compareBySize);
}

DataFileComparator dfc = new DataFileComparator();
Comparator<FileMetadata> comp = dfc.compareBy(folderPresort, tagPresort, fileSortField, !"desc".equals(fileSortOrder));
Collections.sort(fileList, comp);
}

private Boolean isIndexedVersion = null;
Expand Down Expand Up @@ -1863,7 +1853,12 @@ private String init(boolean initFull) {
String nonNullDefaultIfKeyNotFound = "";
protocol = settingsWrapper.getValueForKey(SettingsServiceBean.Key.Protocol, nonNullDefaultIfKeyNotFound);
authority = settingsWrapper.getValueForKey(SettingsServiceBean.Key.Authority, nonNullDefaultIfKeyNotFound);
if (this.getId() != null || versionId != null || persistentId != null) { // view mode for a dataset
String sortOrder = getSortOrder();
if(sortOrder != null) {
FileMetadata.setCategorySortOrder(sortOrder);
}

if (dataset.getId() != null || versionId != null || persistentId != null) { // view mode for a dataset

DatasetVersionServiceBean.RetrieveDatasetVersionResponse retrieveDatasetVersionResponse = null;

Expand Down Expand Up @@ -2245,6 +2240,19 @@ private void displayLockInfo(Dataset dataset) {

}

public String getSortOrder() {
return settingsWrapper.getValueForKey(SettingsServiceBean.Key.CategoryOrder, null);
}

public boolean orderByFolder() {
return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.OrderByFolder, true);
}

public boolean allowUserManagementOfOrder() {
return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.AllowUserManagementOfOrder, false);
}


private Boolean fileTreeViewRequired = null;

public boolean isFileTreeViewRequired() {
Expand All @@ -2267,6 +2275,7 @@ public String getFileDisplayMode() {
}

public void setFileDisplayMode(String fileDisplayMode) {
isPageFlip = true;
if ("Table".equals(fileDisplayMode)) {
this.fileDisplayMode = FileDisplayStyle.TABLE;
} else {
Expand All @@ -2278,13 +2287,6 @@ public boolean isFileDisplayTable() {
return fileDisplayMode == FileDisplayStyle.TABLE;
}

public void toggleFileDisplayMode() {
if (fileDisplayMode == FileDisplayStyle.TABLE) {
fileDisplayMode = FileDisplayStyle.TREE;
} else {
fileDisplayMode = FileDisplayStyle.TABLE;
}
}
public boolean isFileDisplayTree() {
return fileDisplayMode == FileDisplayStyle.TREE;
}
Expand Down Expand Up @@ -2804,6 +2806,24 @@ public void refresh(ActionEvent e) {
refresh();
}


public void sort() {
// This is called as the presort checkboxes' listener when the user is actually
// clicking in the checkbox. It does appear to happen after the setTagPresort
// and setFolderPresort calls.
// So -we know this isn't a pageflip and at this point can update to use the new
// values.
isPageFlip = false;
if (!newTagPresort == tagPresort) {
tagPresort = newTagPresort;
}
if (!newFolderPresort == folderPresort) {
folderPresort = newFolderPresort;
}
sortFileMetadatas(fileMetadatasSearch);
JsfHelper.addSuccessMessage(BundleUtil.getStringFromBundle("file.results.presort.change.success"));
}

public String refresh() {
logger.fine("refreshing");

Expand Down Expand Up @@ -5557,6 +5577,10 @@ public void clearSelection() {
}

public void fileListingPaginatorListener(PageEvent event) {
// Changing to a new page of files - set this so we can ignore changes to the
// presort checkboxes. (This gets called before the set calls for the presorts
// get called.)
isPageFlip=true;
setFilePaginatorPage(event.getPage());
}

Expand Down Expand Up @@ -5673,52 +5697,34 @@ public boolean isSomeVersionArchived() {
return someVersionArchived;
}

private static Date getFileDateToCompare(FileMetadata fileMetadata) {
DataFile datafile = fileMetadata.getDataFile();

if (datafile.isReleased()) {
return datafile.getPublicationDate();
public boolean isTagPresort() {
return this.tagPresort;
}

return datafile.getCreateDate();
}

private static final Comparator<FileMetadata> compareByLabelZtoA = new Comparator<FileMetadata>() {
@Override
public int compare(FileMetadata o1, FileMetadata o2) {
return o2.getLabel().toUpperCase().compareTo(o1.getLabel().toUpperCase());
}
};

private static final Comparator<FileMetadata> compareByNewest = new Comparator<FileMetadata>() {
@Override
public int compare(FileMetadata o1, FileMetadata o2) {
return getFileDateToCompare(o2).compareTo(getFileDateToCompare(o1));
public void setTagPresort(boolean tagPresort) {
// Record the new value
newTagPresort = tagPresort && (null != getSortOrder());
// If this is not a page flip, it should be a real change to the presort
// boolean that we should use.
if (!isPageFlip) {
this.tagPresort = tagPresort && (null != getSortOrder());
}
}
};

private static final Comparator<FileMetadata> compareByOldest = new Comparator<FileMetadata>() {
@Override
public int compare(FileMetadata o1, FileMetadata o2) {
return getFileDateToCompare(o1).compareTo(getFileDateToCompare(o2));
public boolean isFolderPresort() {
return this.folderPresort;
}
};

private static final Comparator<FileMetadata> compareBySize = new Comparator<FileMetadata>() {
@Override
public int compare(FileMetadata o1, FileMetadata o2) {
return (new Long(o1.getDataFile().getFilesize())).compareTo(new Long(o2.getDataFile().getFilesize()));
public void setFolderPresort(boolean folderPresort) {
//Record the new value
newFolderPresort = folderPresort && orderByFolder();
// If this is not a page flip, it should be a real change to the presort
// boolean that we should use.
if (!isPageFlip) {
this.folderPresort = folderPresort && orderByFolder();
}
}
};

private static final Comparator<FileMetadata> compareByType = new Comparator<FileMetadata>() {
@Override
public int compare(FileMetadata o1, FileMetadata o2) {
String type1 = StringUtil.isEmpty(o1.getDataFile().getFriendlyType()) ? "" : o1.getDataFile().getContentType();
String type2 = StringUtil.isEmpty(o2.getDataFile().getFriendlyType()) ? "" : o2.getDataFile().getContentType();
return type1.compareTo(type2);
}
};

public void explore(ExternalTool externalTool) {
ApiToken apiToken = null;
Expand Down
25 changes: 23 additions & 2 deletions src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import edu.harvard.iq.dataverse.util.MarkupChecker;
import edu.harvard.iq.dataverse.util.PersonOrOrgUtil;
import edu.harvard.iq.dataverse.util.BundleUtil;
import edu.harvard.iq.dataverse.util.DataFileComparator;
import edu.harvard.iq.dataverse.DatasetFieldType.FieldType;
import edu.harvard.iq.dataverse.branding.BrandingUtil;
import edu.harvard.iq.dataverse.dataset.DatasetUtil;
Expand Down Expand Up @@ -242,14 +243,34 @@ public List<FileMetadata> getFileMetadatas() {
}

public List<FileMetadata> getFileMetadatasSorted() {
Collections.sort(fileMetadatas, FileMetadata.compareByLabel);

/*
* fileMetadatas can sometimes be an
* org.eclipse.persistence.indirection.IndirectList When that happens, the
* comparator in the Collections.sort below is not called, possibly due to
* https://bugs.eclipse.org/bugs/show_bug.cgi?id=446236 which is Java 1.8+
* specific Converting to an ArrayList solves the problem, but the longer term
* solution may be in avoiding the IndirectList or moving to a new version of
* the jar it is in.
*/
if(!(fileMetadatas instanceof ArrayList)) {
List<FileMetadata> newFMDs = new ArrayList<FileMetadata>();
for(FileMetadata fmd: fileMetadatas) {
newFMDs.add(fmd);
}
setFileMetadatas(newFMDs);
}

DataFileComparator dfc = new DataFileComparator();
Collections.sort(fileMetadatas, dfc.compareBy(true, null!=FileMetadata.getCategorySortOrder(), "name", true));
return fileMetadatas;
}

public List<FileMetadata> getFileMetadatasSortedByLabelAndFolder() {
ArrayList<FileMetadata> fileMetadatasCopy = new ArrayList<>();
fileMetadatasCopy.addAll(fileMetadatas);
Collections.sort(fileMetadatasCopy, FileMetadata.compareByLabelAndFolder);
DataFileComparator dfc = new DataFileComparator();
Collections.sort(fileMetadatasCopy, dfc.compareBy(true, null!=FileMetadata.getCategorySortOrder(), "name", true));
return fileMetadatasCopy;
}

Expand Down
Loading

0 comments on commit bb01984

Please sign in to comment.