fix(ingest): drop empty fields

[anshbansal]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[hsheth2]

might've been better to rename auto_fix_duplicate_schema_field_paths -> auto_fix_schema_field_paths, and put this functionality in the same method

[anshbansal]

airflow failure is being fixed here #11616

[anshbansal]

testQuick passed but it is stuck for some reason. Happening in multiple PRs

• fix(ingestion/glue): manage table names from resource_links from nearest catalog correctly #11578
• feat(ingest/oracle): retire deprecated cx_oracle library #11607

So merging as not related to this PR

[tkdrahn]

I don’t believe we have that pulled into our ingestion code, will take a look.

It looks like that will just drop the field completely from the Schema, that’s not what we want either is it?

[hsheth2]

I’d like to understand the empty column name thing - we had assume that that wasn’t really a valid thing (and it also messes up our UI)

What’s the use case for it - eg how do you query a table/view with an empty column name?

[tkdrahn]

I certainly wouldn’t do it but our catalog team doesn’t own the data sets we’re trying to ingest. We have a large number of Snowflake views at our company that have a column with a blank column name

here is an example Snowflake SQL to demonstrate:

create view example_view as ( select field1 as “”, field2 as “field2” from my_base_table);

select * from example_view where “” = ‘some value’;