Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest/snowflake): speed up snowflake usage query #9899

Merged
merged 3 commits into from
Feb 21, 2024
Merged

feat(ingest/snowflake): speed up snowflake usage query #9899

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b3b5cdf
feat(ingest/snowflake): speed up snowflake usage query
hsheth2 Feb 21, 2024
5889c17
remove inner user join
hsheth2 Feb 21, 2024
aefd0ac
fix
hsheth2 Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_query.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -588,12 +588,15 @@ def usage_per_object_per_time_bucket_for_time_window(
query_id,
query_start_time,
user_name,
NVL(USERS.email, CONCAT(user_name, '{email_domain}')) AS user_email,
-- Construct the email in the query, should match the Python behavior.
-- The user_email is only used by the email_filter_query.
NVL(USERS.email, CONCAT(LOWER(user_name), '{email_domain}')) AS user_email,
{objects_column}
from
snowflake.account_usage.access_history
LEFT JOIN
snowflake.account_usage.users USERS
ON user_name = users.name
WHERE
query_start_time >= to_timestamp_ltz({start_time_millis}, 3)
AND query_start_time < to_timestamp_ltz({end_time_millis}, 3)
Expand Down
Loading