More friendly WARC prefix inside WACZ based on Org slug + Crawl Name / First Seed URL. #1537
Conversation
- supports setting 'WARC_PREFIX' env var in browsertrix crawler (requires crawler 1.0.0-beta.4 or higher) - prefix set to <org slug>-<slug [crawl name | first seed host]> - using either crawl name, if provided, or host name of first seed. both are converted to slug (lowercase alphanum, seperate by dashes) - fixes #412
| ) | ||
| print("Scheduled Crawl Created: " + crawl_id) | ||
|
|
||
| crawl_id, crawljob = self.new_crawl_job_yaml( |
There was a problem hiding this comment.
This just moved from above so that warc_prefix would already be computed.
|
Other thought: we might want to try to cap filename sizes < 255 characters to avoid problems on local filesystems when unzipping the WACZ. Since we have a common suffix length (63 characters from preceding dash before crawl id to file suffix), we could enforce the prefix not being longer than an arbitrary amount of characters - I think maybe 100 would be reasonable? But might get exceeded if there's a long org slug and a long first URL currently. |
It does just use the hostname, not the full URL, and hopefully the name will be fairly short. I'm inclined to wait and see if this becomes an issue at all. |
…/ First Seed URL. (#1537) Supports setting WARC prefix for WARCs inside WACZ to `<org slug>-<slug [crawl name | first seed host]>`. - Prefix set via WARC_PREFIX env var, supported in browsertrix-crawler 1.0.0-beta.4 or higher If crawl name is provided, uses crawl name, other hostname of first seed. The name is 'sluggified', using lowercase alphanum characters separated by dashes. Ex: in an organization called `Default Org`, a crawl of `https://specs.webrecorder.net/` and no name will have WARCs named: `default-org-specs-webrecorder-net-....warc.gz` If the crawl is given the name `SPECS`, the WARCs will be named `default-org-specs-manual-....warc.gz` Fixes #412 in a default way.
Supports setting WARC prefix for WARCs inside WACZ to
<org slug>-<slug [crawl name | first seed host]>.If crawl name is provided, uses crawl name, other hostname of first seed. The name is 'sluggified', using lowercase alphanum characters separated by dashes.
Ex: in an organization called
Default Org, a crawl ofhttps://specs.webrecorder.net/and no name will have WARCs named:default-org-specs-webrecorder-net-....warc.gzIf the crawl is given the name
SPECS, the WARCs will be nameddefault-org-specs-manual-....warc.gzFixes #412 in a default way.