Generalize raster metadata

[brownag]

Following up on #1696 (comment) I did some analysis on removing driver restrictions for reading and writing raster metadata.

It seems that with removing a few conditional statements we can write either .aux.xml or internal metadata for time and units for many more drivers than just GTiff.

I wrote a routine (see this Gist) to attempt to write and read a raster with metadata using all available raster drivers. The .aux.json is deleted prior to read, so any metadata that are read are those written using GDAL SetMetadataItem(), not the terra-specific .aux.json function.

The following is from testing with GDAL 3.10.0 and config option GDAL_PAM_ENABLED=YES (default).

Of 80 raster drivers with apparent write capability, I found 2 that I would omit from the test (MEM and VRT). 10 additional drivers failed to write--some of these are expected, like OpenFileGDB, which does not support write for raster, the others need more specific configuration to make copies of existing data sources or I don't have a good way to test them in this context.

I found 28 drivers can read or write all of the current metadata elements including unit, time, dataset, band user tags. One driver (GPKG) can write all but the band-level user tags. An additional 13 drivers read and write units and time only. This gives a total of 42 drivers that can read and write units and time without significant changes to current code. 32 drivers can read and write dataset user tags, and 31 can do band-level user tags. 55 drivers write some sort of .aux.xml file, of which 50 contain both units and time, and 53 only time. 33 write dataset and band tags to .aux.xml files. So, there appear to be some cases where tags are written but not read.

Is there anywhere I have missed where there may be some logic limiting read/write for these metadata items? I have tried playing with the logic in read_gdal.cpp but so far have only made the above results the same or worse.

terra/src/read_gdal.cpp

Lines 1261 to 1290 in 2a3aa0b

	if ((gdrv=="netCDF") || (gdrv == "HDF5")) {
	char **m = poDataset->GetMetadata();
	if (m) {
	while (*m != nullptr) {
	metadata.push_back(*m++);
	}
	}
	s.set_names_time_ncdf(metadata, bandmeta, msg);
	if (s.srs.is_empty()) {
	bool lat = false;
	bool lon = false;
	for (size_t i=0; i<metadata.size(); i++) {
	if (!lat) lat = metadata[i].find("long_name=latitude");
	if (!lon) lon = metadata[i].find("long_name=longitude");
	}
	if (lon && lat) {
	if (s.srs.set("+proj=longlat", msg)) {
	s.parameters_changed = true;
	}
	}
	}
	} else if (gdrv == "GRIB") {
	s.set_names_time_grib(bandmeta, msg);
	} else if (gdrv == "GTiff") {
	// needs to get its own generic one
	// s.set_names_time_tif(bandmeta, msg);
	}

In general it seems that DATE_TIME is better supported than UNIT, and TIMESTAMP/TIMEUNIT is slightly better supported than DATE_TIME. Possibly of interest to highlight: 5 drivers ("COG", "GPKG", "GTiff", "netCDF", "PCIDSK") support read/write of internal metadata for units and time.

While I think this is an improvement over only writing GDAL metadata with GTiff, it is not as consistent as I would hope. It does seem that the .aux.json file provides some consistency at the cost of being external to GDAL.

I am not sure yet why some drivers don't write .aux.xml or internal metadata, there is probably not one reason, I guess the SetMetadataItem()/GetMetadata() calls etc. are just ignored/not defined in some drivers. It is documented that some drivers do not support them, but it seems like a few of the problem drivers should work better than they do.

I looked into my Zarr anecdote from #1696 (comment) and found that writeRaster()/SetMetadataItem() appear to not write to the internal .zmetadata file, but I could write custom metadata for Zarr with the -mo flag to gdal_translate. While I could write e.g. UNIT metadata manually, it was still not read with rast(). I may investigate a bit more how gdal_translate is getting/setting metadata to try and figure that out.

[rhijmans]

That is excellent. Thank you for being so thorough. Let's go ahead with this.

[brownag]

That is excellent. Thank you for being so thorough. Let's go ahead with this.

OK, I've marked it as ready to review. If you want me to clean anything up let me know, but figured for now I would just comment out the limiting logic