Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sled-agent fails to report datasets in inventory if snapshots are present #7743

Closed
jgallagher opened this issue Mar 6, 2025 · 0 comments · Fixed by #7745
Closed

sled-agent fails to report datasets in inventory if snapshots are present #7743

jgallagher opened this issue Mar 6, 2025 · 0 comments · Fixed by #7745
Assignees
@jgallagher

Comments

@jgallagher
Copy link
Contributor

The dogfood blueprint_rendezvous RPW is reporting that there are 14 debug datasets present in the blueprint but not in inventory collections:

    target blueprint:     be34579a-dc81-4242-91e5-aa3ef7329d91
    inventory collection: dbf7167a-912e-47e1-80c6-193b4d305b6e
    debug_dataset rendezvous counts:
        num_inserted:           0
        num_already_exist:      105
        num_not_in_inventory:   14
        num_tombstoned:         0
        num_already_tombstoned: 1
    crucible_dataset rendezvous counts:
        num_inserted:         0
        num_already_exist:    119
        num_not_in_inventory: 0

Of those 14:

  • 10 of them are from sled 14 (i.e., all of this sled's debug datasets)
  • the other four are spread out amongst four other sleds

Looking at the inventory from sled 14, it's actually reporting no datasets at all:

BRM42220051 # curl http://[fd00:1122:3344:10b::1]:12345/inventory | jq | grep datasets
  "datasets": [],

All the expected datasets do exist in both the omicron-datasets.json ledger and in the output of zfs list.

On a debugging call, we ran the same command sled-agent runs when asked for inventory to get dataset properties and noticed a zone-archives snapshot in the output:

BRM42220051 # zfs get -d 1 -Hpo "name,property,value,source" oxide:uuid,name,avail,used,quota,reservation,compression oxp_b6932bb0-bab8-4876-914a-9c75a600e794 oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794        -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        used    1296441416704   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        quota   0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        reservation     0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794        compression     off     default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       oxide:uuid      800d1758-9312-4b1a-8f02-dc6d644c2a9b    local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       used    849601432576    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       quota   0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       reservation     0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crucible       compression     off     local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  used    446604246528    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  quota   0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  reservation     0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  compression     off     default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  used    446604246528    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  quota   0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  reservation     0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt  compression     off     default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   available       -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   used    11165696        -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   quota   -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   reservation     -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt@zone-archives-oxp_b6932bb0-bab8-4876-914a-9c75a600e794   compression     -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    oxide:uuid      8c4ffb2f-92ec-4d4b-8c20-925ddc552dc0    local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    available       107374081024    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    used    101376  -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    quota   107374182400    local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    reservation     0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/debug    compression     gzip-9  local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     used    965072384       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     quota   0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     reservation     0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     compression     off     local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     used    965072384       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     quota   0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     reservation     0       local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone     compression     off     local
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   used    317879296       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   quota   0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   reservation     0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_crucible_800d1758-9312-4b1a-8f02-dc6d644c2a9b   compression     off     inherited from oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    used    323525632       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    quota   0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    reservation     0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_a3f9e352-3090-4331-89a6-280715627279    compression     off     inherited from oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    oxide:uuid      -       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    name    oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    available       1799156261888   -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    used    323488256       -
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    quota   0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    reservation     0       default
oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone/oxz_propolis-server_e7832898-283f-4965-b94c-9c4b8a1b1081    compression     off     inherited from oxp_b6932bb0-bab8-4876-914a-9c75a600e794/crypt/zone

@andrewjstone noted that sled-agent logs parsing failures, and indeed we found them; e.g.,

17:49:01.587Z WARN SledAgent: Failed to access dataset info within zpool
    err = Failed to parse 'available'
    file = sled-agent/src/sled_agent.rs:1334
    sled_id = 71def415-55ad-46b4-ba88-3ca55d7fb287
    zpool = oxp_b6932bb0-bab8-4876-914a-9c75a600e794

Our parser is expecting available to be a u64, but for snapshots it is -.

I think we were not expecting snapshots to show up in this zfs get call; if that's right, fixing this should be adding -t filesystem to avoid that.

All 14 of the missing debug datasets have similar zone-archives snapshots.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jgallagher jgallagher

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sled-agent: Restrict get_dataset_properties to filesystems and volumes oxidecomputer/omicron
1 participant
@jgallagher