Skip to content

Commit 9b58c88

Browse files
vshshjn7Vishesh Jain
and
Vishesh Jain
authored
[CP][EWT-302]Patch Pool.DEFAULT_POOL_NAME in BaseOperator (apache#8587) (#49)
Open source commit id: b37ce29 Co-authored-by: Vishesh Jain <visheshj@twitter.com>
1 parent d5d0a07 commit 9b58c88

File tree

2 files changed

+19
-7
lines changed

2 files changed

+19
-7
lines changed

UPDATING.md

+17-5
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,18 @@ under the License.
2222
This file documents any backwards-incompatible changes in Airflow and
2323
assists users migrating to a new version.
2424

25+
## CP
26+
27+
### Ability to patch Pool.DEFAULT_POOL_NAME in BaseOperator
28+
It was not possible to patch pool in BaseOperator as the signature sets the default value of pool
29+
as Pool.DEFAULT_POOL_NAME.
30+
While using subdagoperator in unittest(without initializing the sqlite db), it was throwing the
31+
following error:
32+
```
33+
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: slot_pool.
34+
```
35+
Fix for this, https://github.com/apache/airflow/pull/8587
36+
2537
## Airflow 1.10.4
2638

2739
### Python 2 support is going away
@@ -36,12 +48,12 @@ If you have a specific task that still requires Python 2 then you can use the Py
3648

3749
### Changes to GoogleCloudStorageHook
3850

39-
* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
51+
* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
4052
* as a part of this replacement, the `multipart` & `num_retries` parameters for `GoogleCloudStorageHook.upload` method have been deprecated.
41-
53+
4254
The client library uses multipart upload automatically if the object/blob size is more than 8 MB - [source code](https://github.com/googleapis/google-cloud-python/blob/11c543ce7dd1d804688163bc7895cf592feb445f/storage/google/cloud/storage/blob.py#L989-L997). The client also handles retries automatically
4355

44-
* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.
56+
* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.
4557

4658
Updating to `google-cloud-storage >= 1.16` changes the signature of the upstream `client.get_bucket()` method from `get_bucket(bucket_name: str)` to `get_bucket(bucket_or_name: Union[str, Bucket])`. This method is not directly exposed by the airflow hook, but any code accessing the connection directly (`GoogleCloudStorageHook().get_conn().get_bucket(...)` or similar) will need to be updated.
4759

@@ -305,7 +317,7 @@ then you need to change it like this
305317
@property
306318
def is_active(self):
307319
return self.active
308-
320+
309321
### Support autodetected schemas to GoogleCloudStorageToBigQueryOperator
310322

311323
GoogleCloudStorageToBigQueryOperator is now support schema auto-detection is available when you load data into BigQuery. Unfortunately, changes can be required.
@@ -317,7 +329,7 @@ define a schema_fields:
317329
gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
318330
...
319331
schema_fields={...})
320-
332+
321333
or define a schema_object:
322334

323335
gcs_to_bq.GoogleCloudStorageToBigQueryOperator(

airflow/models/baseoperator.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -281,7 +281,7 @@ def __init__(
281281
priority_weight=1, # type: int
282282
weight_rule=WeightRule.DOWNSTREAM, # type: str
283283
queue=configuration.conf.get('celery', 'default_queue'), # type: str
284-
pool=Pool.DEFAULT_POOL_NAME, # type: str
284+
pool=None, # type: str
285285
sla=None, # type: Optional[timedelta]
286286
execution_timeout=None, # type: Optional[timedelta]
287287
on_failure_callback=None, # type: Optional[Callable]
@@ -351,7 +351,7 @@ def __init__(
351351
self.retries = retries if retries is not None else \
352352
int(configuration.conf.get('core', 'default_task_retries', fallback=0))
353353
self.queue = queue
354-
self.pool = pool
354+
self.pool = Pool.DEFAULT_POOL_NAME if pool is None else pool
355355
self.sla = sla
356356
self.execution_timeout = execution_timeout
357357
self.on_failure_callback = on_failure_callback

0 commit comments

Comments
 (0)