Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(fs): Listing only lists directories and not objects #3960

Closed
amunra opened this issue Jan 9, 2024 · 11 comments
Closed

bug(fs): Listing only lists directories and not objects #3960

amunra opened this issue Jan 9, 2024 · 11 comments

Comments

@amunra
Copy link
Contributor

amunra commented Jan 9, 2024

We're using OpenDAL as an abstraction across different services.

We expect things to behave the same across S3, Azure and the filesystem, however we found that it isn't really always the case.

One instance is listing a path.

  • In Azure & S3, listing returns back paths to objects.
  • On the filesystem, objects aren't listed, instead only directories.
@Xuanwo
Copy link
Member

Xuanwo commented Jan 9, 2024

Are you talking about op.list_with(path).recursive(true)?

@Xuanwo
Copy link
Member

Xuanwo commented Jan 9, 2024

Would you like to share the opendal verison and your file tree layout and the code to list them?

@amunra
Copy link
Contributor Author

amunra commented Jan 9, 2024

Java code:

    @NotNull
    public static Set<String> listFiles(BlockingOperator client) {
        return client.list("/").stream()
                .map((entry) -> entry.path)
                .collect(Collectors.toSet());
    }

I'm pretty pressed between tasks at the moment and can't produce a full repro, but we dedicate a fresh new directory in our tests and read and write that directory via OpenDAL exclusively.

At certain points we list to assert side-effects and those assertions fail when using fs rather than S3 or Azure.

We've worked around this by checking for object existence by doing a read and the object is there, so I suspect it's a list issue rather than anything else.

Apologies for not being able to provide more context at this point.

Versions.

The software that writes uses:

# Cargo.toml
opendal = "0.42.0"

The software that tests uses:

<!-- pom.xml -->
<dependency>
    <groupId>org.apache.opendal</groupId>
    <artifactId>opendal-java</artifactId>
    <version>0.42.0</version>
    <scope>test</scope>
</dependency>

@Xuanwo
Copy link
Member

Xuanwo commented Jan 9, 2024

Thanks a lot.

list(/) should return all entires under /. For example:

/
/file_a
/dir_b/file_c

The returning list should be ["file_a", "dir_b/"]. Are you saying that fs only returns dir_b/ but missed file_a? Will this changed after upgrading to our latest 0.44.1?

@bluestreak01
Copy link

it returns list of top-level directories for fs schema, e.g. from your example we see:

/
/file_a
/dir_b

@Xuanwo
Copy link
Member

Xuanwo commented Jan 10, 2024

it returns list of top-level directories for fs schema, e.g. from your example we see:

  • / should not be returned
  • file_a is a file, it's ok to be listed here.
  • dir_b should be present, but it's path should be dir_b/.

I will try to reproduce this and create a behavior test for this.

@Zheaoli
Copy link
Member

Zheaoli commented Jan 10, 2024

The directory structure is

Permissions Size User      Date Modified Name
drwxr-xr-x     - manjusaka 11 1月  01:43 /tmp/opendal
drwxr-xr-x     - manjusaka 11 1月  01:43 ├── demo
.rw-r--r--     0 manjusaka 11 1月  01:43 │  └── abc
.rw-r--r--     0 manjusaka 11 1月  01:43 └── demo2

The java code

       System.out.println( blockingOp().list("opendal/").stream()
           .map((entry) -> entry.path)
           .collect(Collectors.toSet()));

the results is

[opendal/demo2, opendal/demo/]

@Xuanwo
Copy link
Member

Xuanwo commented Feb 5, 2024

Hi @amunra, would you like to try 0.45 for verify?

@amunra
Copy link
Contributor Author

amunra commented Feb 5, 2024

Thanks!

It may take me a little while to get back on this one.

At the moment we're stuck with 0.42.0 since we've encountered some yet-undiagnosed bug with S3 with newer releases which only happens with live S3 instances and not our s3s-fs—based mock infra.

@Xuanwo
Copy link
Member

Xuanwo commented Feb 5, 2024

It may take me a little while to get back on this one.

Got it.

At the moment we're stuck with 0.42.0 since we've encountered some yet-undiagnosed bug with S3 with newer releases which only happens with live S3 instances and not our s3s-fs—based mock infra.

Please let me know if any help we can provide.

@Xuanwo
Copy link
Member

Xuanwo commented Mar 12, 2025

Hi, I'm going to close this issue now. Feel free to reopen it if the problem persists.

@Xuanwo Xuanwo closed this as not planned Won't fix, can't repro, duplicate, stale Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants