Start-up crashing due to Error: root can't be a file (unixfs type: File) [possibly related to recent Kubo 0.34.0 upgrade]

[vorburger]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

third-party binary

Version

$ ipfs version --all
Kubo version: 0.34.0
Repo version: 16
System version: amd64/linux
Golang version: go1.24.1

$ which ipfs
/usr/bin/ipfs

$ rpm -qf /usr/bin/ipfs
kubo-0.34.0-0.1.testing.rp.fc41.taw.x86_64


This is the binary from https://github.com/taw00/ipfs-rpm.

Config

$ ipfs config show
{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Origin": [
        "https://webui.ipfs.io",
        "http://webui.ipfs.io.ipns.localhost:8080"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/webrtc-direct",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/webrtc-direct",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "AutoTLS": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic-v1/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/ipfs/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "DeserializedResponses": null,
    "DisableHTMLErrors": null,
    "ExposeRoutingAPI": null,
    "HTTPHeaders": {},
    "NoDNSLink": false,
    "NoFetch": false,
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWMNTJPcXCAtenjQCYSmGj7DRKzjMYgcPpy3rd9BgRSoER"
  },
  "Import": {
    "CidVersion": null,
    "HashFunction": null,
    "UnixFSChunker": null,
    "UnixFSRawLeaves": null
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "Version": {}
}

Description

See taw00/ipfs-rpm#8; TL;DR :

Mar 22 14:34:36 yara ipfs[138484]: 2025-03-22T14:34:36.531+0100        ERROR        node        node/mount_unix.go:96        error mounting: root can't be a file (unixfs type: File)
Mar 22 14:34:36 yara ipfs[138484]: Error: root can't be a file (unixfs type: File)
Mar 22 14:34:36 yara systemd[1]: ipfsd.service: Main process exited, code=exited, status=1/FAILURE
Mar 22 14:34:36 yara systemd[1]: ipfsd.service: Failed with result 'exit-code'.
Mar 22 14:34:46 yara systemd[1]: ipfsd.service: Scheduled restart job, restart counter is at 884.
Mar 22 14:34:46 yara systemd[1]: Starting ipfsd.service - ipfs daemon...
Mar 22 14:34:46 yara systemd[1]: Started ipfsd.service - ipfs daemon.
Mar 22 14:34:46 yara ipfs[138562]: Initializing daemon...
Mar 22 14:34:46 yara ipfs[138562]: Kubo version: 0.34.0
Mar 22 14:34:46 yara ipfs[138562]: Repo version: 16
Mar 22 14:34:46 yara ipfs[138562]: System version: amd64/linux
Mar 22 14:34:46 yara ipfs[138562]: Golang version: go1.24.1
Mar 22 14:34:46 yara ipfs[138562]: PeerID: 12D3KooWHhSWa5Kz6FBmsM2rENyE3WNz61Yndm83ud8ke95u7Zsp
Mar 22 14:34:46 yara ipfs[138562]: 2025/03/22 14:34:46 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
Mar 22 14:34:47 yara ipfs[138562]: Swarm listening on 100.110.174.7:4001 (TCP+UDP)
Mar 22 14:34:47 yara ipfs[138562]: Swarm listening on 127.0.0.1:4001 (TCP+UDP)
Mar 22 14:34:47 yara ipfs[138562]: Swarm listening on 192.168.1.99:4001 (TCP+UDP)
Mar 22 14:34:47 yara ipfs[138562]: Swarm listening on [::1]:4001 (TCP+UDP)
Mar 22 14:34:47 yara ipfs[138562]: Swarm listening on [fd7a:115c:a1e0::b701:ae07]:4001 (TCP+UDP)
Mar 22 14:34:47 yara ipfs[138562]: Run 'ipfs id' to inspect announced and discovered multiaddrs of this node.
Mar 22 14:34:47 yara ipfs[138562]: RPC API server listening on /ip4/127.0.0.1/tcp/5001
Mar 22 14:34:47 yara ipfs[138562]: WebUI: http://127.0.0.1:5001/webui
Mar 22 14:34:47 yara ipfs[138562]: 2025-03-22T14:34:47.304+0100        ERROR        node        node/mount_unix.go:96        error mounting: root can't be a file (unixfs type: File)
Mar 22 14:34:47 yara ipfs[138562]: Error: root can't be a file (unixfs type: File)
Mar 22 14:34:47 yara systemd[1]: ipfsd.service: Main process exited, code=exited, status=1/FAILURE
Mar 22 14:34:47 yara systemd[1]: ipfsd.service: Failed with result 'exit-code'.
(...)

I'm not entirely sure if this is a problem in this project (IPFS Kubo) something wrong "downstream" (that systemd RPM for Fedora which is graciously maintained by @taw00) - so I'll open issues in both repos.. 😸

Looks like #10340 and #2383 ? Except I'm not trying to upload anything anywhere - just starting it.

My hunch is that this could be related to and caused by (yesterday's!) Kubo 0.34.0 release, which via https://github.com/taw00/ipfs-rpm, I seem to now run on. It may also be a coincidence and totally unrelated - but seemed worth pointing out.

[vorburger]

Does this error mean that it could not --mount-ipfs=${MOUNTPOINT_IPFS} --mount-ipns=${MOUNTPOINT_IPNS} from /usr/lib/systemd/system/ipfsd.service from https://github.com/taw00/ipfs-rpm/blob/master/SOURCES/kubo-contrib/systemd/usr-lib-systemd-system_ipfsd.service ?

Per /etc/sysconfig/ipfsd from https://github.com/taw00/ipfs-rpm/blob/master/SOURCES/kubo-contrib/systemd/etc-sysconfig_ipfsd that's apparently set to MOUNTPOINT_IPFS=/ipfs/ipfsd.service/ipfs & MOUNTPOINT_IPNS=/ipfs/ipfsd.service/ipns (perhaps the error message could print it out, for more clarity?), but it looks fine, to me:

$ ls -al /ipfs/ipfsd.service/
drwxr-x---. 1 ipfs ipfs 16 Mar 21 01:00 ./
drwxrwx---. 1 ipfs ipfs 26 Mar 21 01:00 ../
drwxr-x---. 1 ipfs ipfs  0 Mar 21 01:00 ipfs/
drwxr-x---. 1 ipfs ipfs  0 Mar 21 01:00 ipns/

$ ls -al /ipfs/ipfsd.service/ipfs/
total 0
drwxr-x---. 1 ipfs ipfs  0 Mar 21 01:00 ./
drwxr-x---. 1 ipfs ipfs 16 Mar 21 01:00 ../

$ ls -al /ipfs/ipfsd.service/ipns/
total 0
drwxr-x---. 1 ipfs ipfs  0 Mar 21 01:00 ./
drwxr-x---. 1 ipfs ipfs 16 Mar 21 01:00 ../

[lidel]

@vorburger thank you for reporting this.

It looks like you are using unofficial RPM package from https://github.com/taw00/ipfs-rpm

Before we spend time investigating, are you able to reproduce problem if you use prebuilt binaries attached to https://github.com/ipfs/kubo/releases/latest or when building it locally?

[vorburger]

Before we spend time investigating, are you able to reproduce problem if you use prebuilt binaries attached to https://github.com/ipfs/kubo/releases/latest or when building it locally?

ACK, that makes sense. I'll see if I can reproduce this problem when building it locally from source, and then update here. If needed, I'll add some better debugging logs in the coffee. (But I won't get to this before next weekend.)

[vorburger]

OK yeah so I can reproduce this, with binary built from source at 370bb9f instead of from that RPM, and using the same manually cobbled together startup-up parameters as it would if it ran under systemd from where I'm seeing this:

$ make build
$ cmd/ipfs/ipfs version
ipfs version 0.35.0-dev

$ sudo -g ipfs -u ipfs IPFS_PATH=/var/lib/ipfs/repo cmd/ipfs/ipfs daemon --mount --mount-ipfs=/ipfs/ipfsd.service/ipfs --mount-ipns=/ipfs/ipfsd.service/ipns
Initializing daemon...
Kubo version: 0.35.0-dev-370bb9fec
Repo version: 16
System version: amd64/linux
Golang version: go1.24.1
PeerID: 12D3KooWHhSWa5Kz6FBmsM2rENyE3WNz61Yndm83ud8ke95u7Zsp
2025/03/29 19:51:12 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
Swarm listening on 100.110.174.7:4001 (TCP+UDP)
Swarm listening on 127.0.0.1:4001 (TCP+UDP)
Swarm listening on 192.168.1.99:4001 (TCP+UDP)
Swarm listening on [::1]:4001 (TCP+UDP)
Swarm listening on [fd7a:115c:a1e0::b701:ae07]:4001 (TCP+UDP)
Run 'ipfs id' to inspect announced and discovered multiaddrs of this node.
RPC API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
2025-03-29T19:51:13.587+0100    ERROR   node    node/mount_unix.go:96   error mounting: root can't be a file (unixfs type: File)

Error: root can't be a file (unixfs type: File)

I'll see if I can debug it a bit more, using this...

[vorburger]

With #10770 I was able to see: 2025-03-29T20:10:35.959+0100 ERROR node node/mount_unix.go:96 error mounting IPNS /ipfs/ipfsd.service/ipns for IPFS /ipfs/ipfsd.service/ipfs: root can't be a file (unixfs type: File) and that at first lead me to think that it may be stupid error, like /ipfs/ipfsd.service/ipns or /ipfs/ipfsd.service/ipfs being a file instead of a directory - but that's actually not at all what this problem is about; as already seen by the ls -al shown above earlier.

I'm guess this comes from here? From my (limited) understanding, that would presumably indicate having a "corrupt" MFS? Understanding that, indeed doing this "fixes" (works around) it, of course:

sudo mv /var/lib/ipfs/repo /var/lib/ipfs/repo.issue-10762

Having figure this out is "good enough" to unblock me, and I won't further pursue this myself.

I'll have to leave it up to maintainers if this condition should be handled more gracefully, or if this is "expected" and "working as intended". (Let me know if you would like me to share my /var/lib/ipfs/repo somewhere for your debugging, if that's of any interest. There is nothing confidential in it. It's 211 MB. I've just played around with it, this is not "production" data.)

[gammazero]

triage note: will investigate further to determine necessary action