Org-wide filename template for created/downloaded/ WARC/WACZ-files

[Klindten]

1. Input/template where you can define how files are named. For everyone, ie. easy
2. Possibility to add warc-field for user - like in Conifer-service

at KB we need to name files webrec_"collectionname" (added date can be fine). Also as some search-discovery tools, like SolrWayback, have case-sensitive search, it could be great to have a way to customize which characters are allowed in the WARC-file name.

[ldko]

I would also find it helpful for consistency with other WARCs we process at UNT to be able to name the WARC files created with a configured pattern akin to in pywb where in config.yml you can configure:

recorder:
   filename_template: UNT-{timestamp}-{hostname}-{random}.warc.gz

[Klindten]

Any news on this? We still need this functionality to test more/go in production.

[Shrinks99]

@Klindten Not much to report unfortunately. We're busy working on QA features! As this is your issue, when it gets updated you'll be the first to know. ;)

I didn't want to completely leave you hanging though, and admitidly this has also been taking up space in my head so I took a few minutes today to create a mockup of where this might fit into the Org Settings page.

Would something like this meet your needs? What fields are required? What might I be missing?

Also worth noting that we may go with a simpler text-based customization system initially as that might be faster to get out the door!

[Klindten]

@Shrinks99 Thanks for getting back to me.

I think we should make a short meeting with you, Colin, Tue, and me asap- this will be easier than ping-pong here. I'll try to find a good time. Where are you based - so it´ll not be too early? Do other people from Webrec need to be in on this

Here´s our initial thoughts:

filenaming template should be for both wacz and warcfiles:

prefix-browsertrix_crawlname-browsertrix_created_by(initials?)-browsertrix_crawltimestamp(YYYYMMDDHHSS).warc.gz

prefix-browsertrix_crawlname-browsertrix_created_by(initials?)-browsertrix_crawltimestamp(YYYYMMDDHHSS).wacz.gz

e.g

webrec-tv2_tema_taleban_afganistan-jens_moeller-202308181556.warc.gz

webrec-tv2_tema_taleban_afganistan-jens_moeller-202308181556.wacz.gz

only prefix is customized/input by user (but once and for all)
(prefix is on an organisational level and should be set automatically)

Could also be. CSR: netarkivet_${crawl_name}${username}${timestamp}.warc.gz

[ikreymer]

The proposed solution for WARCs is implemented in #1537. The WARCs will start with the prefix: <org slug>-<slug [crawl name | first seed host]>-.

WACZs are stored in S3 following a specific convention, so a bit harder to change, but very easy to simply rename after downloading / download to a custom named file simply by doing 'Save Link As...'. Additionally, collections offer a way to have custom WACZ name based on the collection and are the recommended approach for curation.

[Klindten]

Graveyard question.

• In archiveweb.page you can rename your crawl after crawl, before download (thus giving the final name to the WARC-files). Sometimes this is handy if you started crawling with a not so descriptive name.

• In Btrix you can change name of crawl under metadata tab while "Running" and "Waiting".

• In "Archived Items" the name can´t be changed. It can be changed for new crawls though. Would it possible to do it the same as in AWP, to be able to change crawlname after crawl. I suspect there are good reasons for the current procedure:-)