[Managed Iceberg] unbounded source #33504

ahmedabu98 · 2025-01-06T18:16:10Z

Unbounded (streaming) source for Managed Iceberg.

See design doc for high level overview: https://s.apache.org/beam-iceberg-incremental-source

…erg_streaming_source

github-actions · 2025-01-30T22:06:51Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

ahmedabu98 · 2025-02-03T15:51:02Z

R: @kennknowles
R: @regadas

Can y'all take a look? I still have to write some tests, but it's at a good spot for a first round of reviews. I ran a bunch of pipelines (w/Legacy DataflowRunner) at different scales and the throughput/scalability looks good.

github-actions · 2025-02-03T15:52:21Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

…erg_streaming_source

kennknowles

Overall, I think all the pieces are in the right place. Just a question about why an SDF is the way it is and a couple code-level comments.

This seems like something you want to test a lot of different ways before it gets into a release. Maybe get another set of eyes like @chamikaramj or @Abacn too. But I'm approving and leaving to your judgment.

sdks/java/io/iceberg/bqms/build.gradle

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SnapshotRange.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadTask.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadTaskDescriptor.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromGroupedTasks.java

kennknowles

Wait actually I forgot I want to have the discussion about the high level toggle between incremental scan source and bounded source.

…erg_streaming_source

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromGroupedTasks.java

…erg_streaming_source

ahmedabu98 · 2025-03-05T18:37:01Z

@chamikaramj this is ready for another review

…erg_streaming_source

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IncrementalScanSource.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromTasks.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WatchForSnapshots.java

chamikaramj

Thanks. LGTM.

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java

sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java

…remove window step; add --strea ming=true validation; add IO links to Managed java doc

codecov · 2025-03-11T17:21:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.32%. Comparing base (c1d0fa4) to head (fce87dc).
Report is 17 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #33504      +/-   ##
============================================
+ Coverage     56.28%   56.32%   +0.03%     
  Complexity     3286     3286              
============================================
  Files          1166     1172       +6     
  Lines        178704   178936     +232     
  Branches       3398     3398              
============================================
+ Hits         100591   100786     +195     
- Misses        74860    74897      +37     
  Partials       3253     3253

Flag	Coverage Δ
python	`81.29% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…erg_streaming_source

…Getsize

…Getsize; add resilience to watch snapshots transform

chamikaramj · 2025-03-19T05:50:22Z

LGTM. Thanks!

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/CreateReadTasksDoFn.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromTasks.java

scwhittle · 2025-03-19T12:11:45Z

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadUtils.java

+/** Helper class for source operations. */
+public class ReadUtils {
+  // default is 8MB. keep this low to avoid overwhelming memory
+  static final int MAX_FILE_BUFFER_SIZE = 1 << 18; // 256KB


might want this to be configurble or pipeline option.

I'd rather we wait until there's a need to expose it

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableCache.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WatchForSnapshots.java

scwhittle

Thanks!

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableCache.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadUtils.java

ahmedabu98 · 2025-03-20T15:08:45Z

Thanks for the review @scwhittle! If all else is good, I'll merge it when tests pass

…u98/beam into iceberg_streaming_source

initial

Loading
Loading status checks…

bb87511

github-actions bot added java io labels Jan 6, 2025

ahmedabu98 marked this pull request as draft January 6, 2025 18:16

ahmedabu98 added 7 commits January 7, 2025 15:50

let CombinedScanTask do splitting (based on Parquet row groups)

Loading
Loading status checks…

853de4d

perf improv

Loading
Loading status checks…

69fd988

create one read task descriptor per snapshot range

Loading
Loading status checks…

da2f33f

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

73c8992

…erg_streaming_source

some improvements

Loading
Loading status checks…

81ca709

use GiB for streaming, Redistribute for batch; update docs

Loading
Loading status checks…

e319d76

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

Loading
Loading status checks…

c25cd75

…erg_streaming_source

ahmedabu98 marked this pull request as ready for review January 30, 2025 21:09

use static value

Loading
Loading status checks…

af1ec85

ahmedabu98 added 3 commits February 3, 2025 16:27

add some test

Loading
Loading status checks…

f5d3268

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

df40239

…erg_streaming_source

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

43ab88f

…erg_streaming_source

kennknowles approved these changes Feb 5, 2025

View reviewed changes

kennknowles requested changes Feb 5, 2025

View reviewed changes

ahmedabu98 added 3 commits February 7, 2025 08:32

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

20db0ee

…erg_streaming_source

add a java doc; don't use static block to create coder

Loading
Loading status checks…

622625f

spotless

Loading
Loading status checks…

4c25d3f

kennknowles approved these changes Feb 11, 2025

View reviewed changes

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java Outdated Show resolved Hide resolved

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromGroupedTasks.java Outdated Show resolved Hide resolved

ahmedabu98 added 2 commits February 13, 2025 12:35

add options: from/to timestamp, starting strategy, and streaming toggle

Loading
Loading status checks…

8666166

trigger integration tests

Loading
Loading status checks…

297c309

github-actions bot added the build label Feb 13, 2025

ahmedabu98 added 2 commits February 13, 2025 13:47

small test fix

Loading
Loading status checks…

5e3a2cc

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

8b131fd

…erg_streaming_source

add another test for read utils

Loading
Loading status checks…

7dbf3e1

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

5263a13

…erg_streaming_source

scwhittle requested changes Mar 6, 2025

View reviewed changes

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WatchForSnapshots.java Outdated Show resolved Hide resolved

chamikaramj reviewed Mar 6, 2025

View reviewed changes

ahmedabu98 mentioned this pull request Mar 7, 2025

[Task]: Add Python AfterSynchronizedProcessingTime trigger and add an Iceberg CDC streaming read test #34212

Open

17 tasks

ahmedabu98 added 4 commits March 7, 2025 10:56

use cached schemas

Loading
Loading status checks…

40fe4ab

watermark based on snapshot timestamp; remove infinite allowed skew; …

Loading
Loading status checks…

db3c570

…remove window step; add --strea ming=true validation; add IO links to Managed java doc

remove check

Loading
Loading status checks…

7078f20

remove watermark support; add a counter and some helpful logging

Loading
Loading status checks…

fce87dc

ahmedabu98 added 4 commits March 11, 2025 17:56

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

adbbfaf

…erg_streaming_source

change metric name

Loading
Loading status checks…

30f3ced

use Redistribute for unbounded; make ReadFromTasks an SDF to leverage @…

Loading
Loading status checks…

078b63a

…Getsize; add resilience to watch snapshots transform

remove CDC schema and output the data record normally

Loading
Loading status checks…

941e8aa

scwhittle requested changes Mar 19, 2025

View reviewed changes

ahmedabu98 added 2 commits March 19, 2025 18:36

use LoadingCache; leverage iceberg bin-packing; address other comments

Loading
Loading status checks…

44923f3

spotless

Loading
Loading status checks…

1c0f7d7

scwhittle reviewed Mar 20, 2025

View reviewed changes

ahmedabu98 force-pushed the iceberg_streaming_source branch from 1f47712 to 1c0f7d7 Compare March 20, 2025 12:18

validate catalog cache consistency; added a TODO

Loading
Loading status checks…

7f12282

Merge branch 'master' into iceberg_streaming_source

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

17816bd

github-actions bot removed the build label Mar 20, 2025

ahmedabu98 added 2 commits March 20, 2025 14:23

trigger integration tests

24d1782

Merge branch 'iceberg_streaming_source' of https://github.com/ahmedab…

Loading
Loading status checks…

3b8bac1

…u98/beam into iceberg_streaming_source

github-actions bot added the build label Mar 20, 2025

ahmedabu98 merged commit 332fe03 into apache:master Mar 20, 2025
110 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Managed Iceberg] unbounded source #33504

[Managed Iceberg] unbounded source #33504

ahmedabu98 commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025

ahmedabu98 commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

kennknowles left a comment

kennknowles left a comment

ahmedabu98 commented Mar 5, 2025

chamikaramj left a comment

codecov bot commented Mar 11, 2025

chamikaramj commented Mar 19, 2025

scwhittle Mar 19, 2025

ahmedabu98 Mar 19, 2025

scwhittle left a comment

ahmedabu98 commented Mar 20, 2025

[Managed Iceberg] unbounded source #33504

[Managed Iceberg] unbounded source #33504

Conversation

ahmedabu98 commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 30, 2025

ahmedabu98 commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

kennknowles left a comment

Choose a reason for hiding this comment

kennknowles left a comment

Choose a reason for hiding this comment

ahmedabu98 commented Mar 5, 2025

chamikaramj left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 11, 2025

Codecov Report

chamikaramj commented Mar 19, 2025

scwhittle Mar 19, 2025

Choose a reason for hiding this comment

ahmedabu98 Mar 19, 2025

Choose a reason for hiding this comment

scwhittle left a comment

Choose a reason for hiding this comment

ahmedabu98 commented Mar 20, 2025

ahmedabu98 commented Jan 6, 2025 •

edited

Loading