Code

Rework crawl page migration#2412

Code

Merged

ikreymer merged 70 commits intomainwebrecorder/browsertrix:mainfrom issue-2406-crawl-migration-reworkwebrecorder/browsertrix:issue-2406-crawl-migration-reworkCopy head branch name to clipboardFeb 20, 2025

+790-526

Commits

Commits on Feb 18, 2025

• migration fixes:

  Show description for 05a7419
  

  ikreymer

  authored and

  tw4l

  committed
  05a7419

  Copy full SHA for 05a7419

• bump version to 1.14.0-beta.2

  

  ikreymer

  authored and

  tw4l

  committed
  0ae4291

  Copy full SHA for 0ae4291

• bump to 1.14.0-beta.3

  Show description for feb4f09
  

  ikreymer

  authored and

  tw4l

  committed
  feb4f09

  Copy full SHA for feb4f09

• typo fix

  

  ikreymer

  authored and

  tw4l

  committed
  16dd7a4

  Copy full SHA for 16dd7a4

• empty filenames migration: use existing bulk re_add method for faster migration

  

  ikreymer

  authored and

  tw4l

  committed
  9ab4dac

  Copy full SHA for 9ab4dac

• bump to 1.14.0-beta.4

  

  ikreymer

  authored and

  tw4l

  committed
  20b6a73

  Copy full SHA for 20b6a73

• Add version to BaseCrawl, set to 2 for new crawls/uploads

  Show description for b57e4fb
  tw4l

  committed
  b57e4fb

  Copy full SHA for b57e4fb

• Use optimize pages background job in 0042

  Show description for e390ba3
  tw4l

  committed
  e390ba3

  Copy full SHA for e390ba3

• Hacky workaroud: Pass default org to retry_background_job

  Show description for 4ea162f
  tw4l

  committed
  4ea162f

  Copy full SHA for 4ea162f

• Make sure job failure emails are sent even if no oid

  tw4l

  committed
  b67e2c7

  Copy full SHA for b67e2c7

• Fix typing error

  tw4l

  committed
  bd2e2c2

  Copy full SHA for bd2e2c2

• Only include replay optimization fields if all pages are optimized

  Show description for eb455a1
  tw4l

  committed
  eb455a1

  Copy full SHA for eb455a1

• Set isMigrating at same time as pulling next crawl

  tw4l

  committed
  bef7a18

  Copy full SHA for bef7a18

• Redo background_jobs typing, add superuser non-org list endpoint

  tw4l

  committed
  4f90555

  Copy full SHA for 4f90555

• Remove crawl_type from optimize_pages bg job

  tw4l

  committed
  25c8244

  Copy full SHA for 25c8244

• Handle running crawls in page optimization

  tw4l

  committed
  1891ef2

  Copy full SHA for 1891ef2

• Remove isMigrating filter from running crawl check

  tw4l

  committed
  b2d97f0

  Copy full SHA for b2d97f0

• Fix bg job retry

  tw4l

  committed
  ccd7f1e

  Copy full SHA for ccd7f1e

• Cast job to right type in retry

  tw4l

  committed
  19d2547

  Copy full SHA for 19d2547

• Add API endpoint to launch migrate crawls job

  tw4l

  committed
  7bc3397

  Copy full SHA for 7bc3397

• Use SuccessResponseId model to include job id in response

  tw4l

  committed
  538d4a3

  Copy full SHA for 538d4a3

• Add first draft of upgrade notes in docs

  tw4l

  committed
  7d27feb

  Copy full SHA for 7d27feb

• cleanup:

  Show description for 821c02d
  ikreymer

  committed
  821c02d

  Copy full SHA for 821c02d

• add logging to optimize pages job

  ikreymer

  committed
  d311692

  Copy full SHA for d311692

• logging

  ikreymer

  committed
  07d3c25

  Copy full SHA for 07d3c25

Commits on Feb 19, 2025

• check for empty result

  ikreymer

  committed
  9fd26bf

  Copy full SHA for 9fd26bf

• version: bump to 1.14.0-beta.5

  ikreymer

  committed
  39ab851

  Copy full SHA for 39ab851

• fix response model

  ikreymer

  committed
  f71a89e

  Copy full SHA for f71a89e

• add retries for add_crawl_pages_to_db_from_wacz()

  Show description for 533f78c
  ikreymer

  committed
  533f78c

  Copy full SHA for 533f78c

• back to single background_job.yaml, just pass scale as param

  ikreymer

  committed
  25c6e3a

  Copy full SHA for 25c6e3a

• fix parallelism setting

  ikreymer

  committed
  b4dcfd4

  Copy full SHA for b4dcfd4

• retry at the wacz level, not crawl

  ikreymer

  committed
  35b7883

  Copy full SHA for 35b7883

• bg jobs: make single crawl readd a background job as well, if crawl_id is provided,

  Show description for ed66a26
  ikreymer

  committed
  ed66a26

  Copy full SHA for ed66a26

• add sort

  ikreymer

  committed
  358bb51

  Copy full SHA for 358bb51

• better call

  ikreymer

  committed
  e9f070b

  Copy full SHA for e9f070b

• logging

  ikreymer

  committed
  e94f8be

  Copy full SHA for e94f8be

• ensure pages are streamed from each wacz on demand

  ikreymer

  committed
  01431d7

  Copy full SHA for 01431d7

• ignore dupes on insertMany

  ikreymer

  committed
  a486571

  Copy full SHA for a486571

• attempt dedup within wacz

  ikreymer

  committed
  d0c274e

  Copy full SHA for d0c274e

• ignore dupe errors?

  ikreymer

  committed
  5d949aa

  Copy full SHA for 5d949aa

• update logging, add crawl_id + url index

  ikreymer

  committed
  18301e3

  Copy full SHA for 18301e3

• lint fix

  ikreymer

  committed
  0936ccf

  Copy full SHA for 0936ccf

• optimize pages: don't use $facet, don't compute actual total

  ikreymer

  committed
  8742657

  Copy full SHA for 8742657

• add index for default sort

  ikreymer

  committed
  0c805f1

  Copy full SHA for 0c805f1

• remove uniquge page counts from org metrics

  ikreymer

  committed
  d379570

  Copy full SHA for d379570

• update model

  ikreymer

  committed
  6ad27e1

  Copy full SHA for 6ad27e1

• remove resources from collection list

  ikreymer

  committed
  b299941

  Copy full SHA for b299941

• optimize coll /replay.json

  Show description for 06e193b
  ikreymer

  committed
  06e193b

  Copy full SHA for 06e193b

• optimize page snapshot query:

  Show description for 1dafa5b
  ikreymer

  committed
  1dafa5b

  Copy full SHA for 1dafa5b

• crawl /replay.json optimization:

  Show description for c95022a
  ikreymer

  committed
  c95022a

  Copy full SHA for c95022a

• cleanup

  ikreymer

  committed
  e3e3651

  Copy full SHA for e3e3651

• fix typo

  ikreymer

  committed
  0efe822

  Copy full SHA for 0efe822

• lint

  ikreymer

  committed
  4d6b21c

  Copy full SHA for 4d6b21c

• Delay rendering components doing data fetching in collection settings dialog until open (#2414)

  Show description for 77caa40
  emma-sg

  authored
  77caa40

  Copy full SHA for 77caa40

• optimize page snapshots, add non-group alternative

  ikreymer

  committed
  d6e76c4

  Copy full SHA for d6e76c4

Commits on Feb 20, 2025

• fix page_count update?

  ikreymer

  committed
  85d4c6f

  Copy full SHA for 85d4c6f

• tweak page snapshot sort order

  ikreymer

  committed
  bf6f24b

  Copy full SHA for bf6f24b

• use gte

  ikreymer

  committed
  f2ce698

  Copy full SHA for f2ce698

• readd unquote

  ikreymer

  committed
  2861292

  Copy full SHA for 2861292

• lint fix

  ikreymer

  committed
  c639e25

  Copy full SHA for c639e25

• tweak

  ikreymer

  committed
  837b35c

  Copy full SHA for 837b35c

• more cleanup:

  Show description for 9e093b9
  ikreymer

  committed
  9e093b9

  Copy full SHA for 9e093b9

• test fix

  ikreymer

  committed
  7042d16

  Copy full SHA for 7042d16

• test fix

  ikreymer

  committed
  15a5332

  Copy full SHA for 15a5332

• Update backend/btrixcloud/crawlmanager.py

  Show description for 3731172
  

  ikreymer

  and

  tw4l

  authored
  3731172

  Copy full SHA for 3731172

• Pages Query: use $text search or prefix search (#2415)

  Show description for 931e351
  

  ikreymer

  and

  tw4l

  authored
  931e351

  Copy full SHA for 931e351

• Fix typo in response

  tw4l

  committed
  6e70662

  Copy full SHA for 6e70662

• Add test asserts for crawl/upload version field

  tw4l

  committed
  fb9b65b

  Copy full SHA for fb9b65b

• remove dupe index

  ikreymer

  committed
  5713f88

  Copy full SHA for 5713f88

• set pages migration job to run at scale 1 by default

  Show description for 967d2bd
  ikreymer

  committed
  967d2bd

  Copy full SHA for 967d2bd