Updating worker concurrency doesn't reflect in the response or UI #1267

shri-astro · 2023-06-15T13:32:44Z

While attempting to update the worker-queue's concurrency through astro deployment worker-queue update command, using the flag --concurrency int to alter the concurrency of a given astro machine doesn't change the concurrency value for the updated worker queue.
However, the astromachine gets updated successfully.

shrividyahegde@Shrividyas-MacBook-Pro Documents % astro deployment worker-queue update --concurrency 30 
Select a Deployment
 #     DEPLOYMENT NAME                       RELEASE NAME                     DEPLOYMENT ID                  DAG DEPLOY ENABLED     
 1     gcp_test_standard                     planetoidal-gegenschein-1614     clix117rh103281jxa9sgyorxg     false                  
 2     qwerty                                barren-parallax-1415             clix15tmm117561iyyf65j7ghx     false                  
 3     gcp_test_standard_through_inspect     ultraviolet-ionosphere-7533      clix47lns235701iz1ddgrjtka     false                  
 4     gcp_standard                          newtonian-ionization-1282        clix5x8nv342781jxabusoal5z     false                  

> 4
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU        Memory     
 1     a5              1 vCPU     2Gi        
 2     a10             2 vCPU     4Gi        
 3     a20             4 vCPU     8Gi        

> 3

 #     WORKER QUEUE     ISDEFAULT     ID                             
 1     default          true          clix5x8nv342801jxa5p5nrvd1     
 2     non-default      false         clix6dvkn357911jxa8p6jcif0     

> 2

Are you sure you want to update the non-default worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y
worker queue non-default for gcp_standard in clix0m97m001i01oiqr98vqb5 workspace updated
shrividyahegde@Shrividyas-MacBook-Pro Documents % astro deployment inspect                             
Select a Deployment
 #     DEPLOYMENT NAME                       RELEASE NAME                     DEPLOYMENT ID                  DAG DEPLOY ENABLED     
 1     gcp_test_standard                     planetoidal-gegenschein-1614     clix117rh103281jxa9sgyorxg     false                  
 2     qwerty                                barren-parallax-1415             clix15tmm117561iyyf65j7ghx     false                  
 3     gcp_test_standard_through_inspect     ultraviolet-ionosphere-7533      clix47lns235701iz1ddgrjtka     false                  
 4     gcp_standard                          newtonian-ionization-1282        clix5x8nv342781jxabusoal5z     false                  

> 4
deployment:
    configuration:
        name: gcp_standard
        description: ""
        runtime_version: 8.4.0
        dag_deploy_enabled: false
        ci_cd_enforcement: true
        scheduler_size: small
        is_high_availability: false
        executor: CeleryExecutor
        scheduler_au: 10
        scheduler_count: 1
        cluster_name: us-central1
        workspace_name: Test Automated Workspace
        deployment_type: HOSTED_SHARED
        cloud_provider: gcp
        region: us-central1
    worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 20
          worker_type: a20
        - name: non-default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 20
          worker_type: a20
    metadata:
        deployment_id: clix5x8nv342781jxabusoal5z
        workspace_id: clix0m97m001i01oiqr98vqb5
        cluster_id: us-central1
        release_name: N/A
        airflow_version: 2.6.1
        current_tag: 8.4.0
        status: UNKNOWN
        created_at: 2023-06-15T13:14:20.827Z
        updated_at: 2023-06-15T13:28:15.54Z
        deployment_url: cloud.astronomer-dev.io/clix0m97m001i01oiqr98vqb5/deployments/clix5x8nv342781jxabusoal5z/analytics
        webserver_url: org-shri-hosted-dedicated-1.astronomer-dev.run/dusoal5z
        workload_identity: astro-newtonian-ionization-128@astro-gcp-data-plane-dev.iam.gserviceaccount.com

What CLI Version did you experience this bug?
1.16.1

This CLI bug is related to which Astronomer Platform?

Astro

📸 Screenshots

The text was updated successfully, but these errors were encountered:

kushalmalani · 2023-06-22T17:22:24Z

@shri-astro To test this on latest main

shri-astro · 2023-06-22T18:37:39Z

Able to set the worker concurrency now but along with a few observations:

The initial range mentioned for worker concurrency is misleading

shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment worker-queue update --concurrency 90 
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME                DEPLOYMENT ID                   DAG DEPLOY ENABLED     
 1     test                galactic-eclipse-4067       clj72yh7w1174401u2kvvqgab9u     false                  
 2     nknk                sidereal-protostar-7638     clj73oa4y1355441uvihu17g8o5     false                  

> 1
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU        Memory     
 1     a5              1 vCPU     2Gi        
 2     a10             2 vCPU     4Gi        
 3     a20             4 vCPU     8Gi        

> 3

 #     WORKER QUEUE     ISDEFAULT     ID                              
 1     default          true          clj72yh7w1174431u2kpouibanf     

> 1
Error: worker queue option is invalid: worker concurrency must be between 1 and 64
shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment worker-queue update --concurrency 64
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME                DEPLOYMENT ID                   DAG DEPLOY ENABLED     
 1     test                galactic-eclipse-4067       clj72yh7w1174401u2kvvqgab9u     false                  
 2     nknk                sidereal-protostar-7638     clj73oa4y1355441uvihu17g8o5     false                  

> 1
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU        Memory     
 1     a5              1 vCPU     2Gi        
 2     a10             2 vCPU     4Gi        
 3     a20             4 vCPU     8Gi        

> 3

 #     WORKER QUEUE     ISDEFAULT     ID                              
 1     default          true          clj72yh7w1174431u2kpouibanf     

> 1

Are you sure you want to update the default worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y
Error: The worker concurrency must be between 1 and 60 (Requested: 64).

kushalmalani · 2023-06-23T15:37:06Z

@shri-astro Can you take another look at this when you can? You would need to pull latest

shri-astro · 2023-06-26T09:16:10Z

On testing the latest main version,it was observed that for a5 the user is not able to set a concurrency within 1-5, the lower limit starts as 5

Error: worker queue option is invalid: worker concurrency must be between 5 and 15

UI allows the update from 1-15.

Similarly for a20:

Error: worker queue option is invalid: worker concurrency must be between 20 and 60

and A10:

Error: worker queue option is invalid: worker concurrency must be between 10 and 30

setting the upper limits like , 15 as concurrency for a5, 30 for a10 and 60 for a20 is working fine.

kushalmalani · 2023-06-27T16:41:14Z

@shri-astro Fixed this in latest main

shri-astro · 2023-06-27T16:57:31Z

Retested and able to update the worker concurrency but the messages displayed while trying outer range of concurrency for a given worker type still displays a wrong lower boundary value.
i.e, updating a a5 by worker concurrency of 120 , displays the below message:

Error: worker queue option is invalid: worker concurrency must be between 5 and 15

However, user is able to update the worker concurrency in this case to 2 as well.(any value less than 5)

shri-astro · 2023-06-27T16:58:28Z

Interesting case being updating of concurrency of a given worker queue to 0 displays a success message but functionally doesn't change anything in the deployment:

shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment worker-queue update --concurrency 0  
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME                 DEPLOYMENT ID                   DAG DEPLOY ENABLED     
 1     test                true-synodic-0298            cljebq2bf2921391uwqyxrdyx8n     true                   
 2     test health         empty-scintillation-8984     cljeitgb8100681uwwbnwpi5tp      false                  
 3     test111             planetary-gyroscope-4233     cljeiv9uo104381uwwcja9z7a2      true                   

> 2
No worker type was specified. Select the worker type to use
 #     WORKER TYPE     CPU        Memory     
 1     a5              1 vCPU     2Gi        
 2     a10             2 vCPU     4Gi        
 3     a20             4 vCPU     8Gi        
 4     a40             8 vCPU     16Gi       

> 2

 #     WORKER QUEUE     ISDEFAULT     ID                             
 1     default          true          cljeitgb8100721uwwum3nvwlj     

> 1

Are you sure you want to update the default worker queue? If there are any tasks in your DAGs assigned to this worker queue, the tasks might get stuck in a queued state and fail to execute (y/n) y
worker queue default for test health in clix0m97m001i01oiqr98vqb5 workspace updated
shrividyahegde@Shrividyas-MacBook-Pro test_dep % ./astro deployment inspect                              
Select a Deployment
 #     DEPLOYMENT NAME     RELEASE NAME                 DEPLOYMENT ID                   DAG DEPLOY ENABLED     
 1     test                true-synodic-0298            cljebq2bf2921391uwqyxrdyx8n     true                   
 2     test health         empty-scintillation-8984     cljeitgb8100681uwwbnwpi5tp      false                  
 3     test111             planetary-gyroscope-4233     cljeiv9uo104381uwwcja9z7a2      true                   

> 2
deployment:
    configuration:
        name: test health
        description: ""
        runtime_version: 8.6.0
        dag_deploy_enabled: false
        ci_cd_enforcement: false
        scheduler_size: small
        is_high_availability: false
        executor: CeleryExecutor
        scheduler_au: 10
        scheduler_count: 1
        cluster_name: us-central1
        workspace_name: Test Automated Workspace
        deployment_type: HOSTED_SHARED
        cloud_provider: gcp
        region: us-central1
    worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 3
          worker_type: a10
    metadata:
        deployment_id: cljeitgb8100681uwwbnwpi5tp
        workspace_id: clix0m97m001i01oiqr98vqb5
        cluster_id: us-central1
        release_name: N/A
        airflow_version: 2.6.2
        current_tag: 8.6.0
        status: DEPLOYING
        created_at: 2023-06-27T16:47:24.116Z
        updated_at: 2023-06-27T16:54:18.648Z
        deployment_url: cloud.astronomer-dev.io/clix0m97m001i01oiqr98vqb5/deployments/cljeitgb8100681uwwbnwpi5tp/analytics
        webserver_url: org-shri-hosted-dedicated-1.astronomer-dev.run/dnwpi5tp
        workload_identity: astro-empty-scintillation-8984@astro-gcp-data-plane-dev.iam.gserviceaccount.com

shri-astro · 2023-06-28T07:27:53Z

Resolved.
retest results:

Error: Worker concurrency cannot be 0. Minimum value starts from 1
Error: worker queue option is invalid: worker concurrency must be between 1 and 15
Error: worker queue option is invalid: worker concurrency must be between 1 and 30
Error: worker queue option is invalid: worker concurrency must be between 1 and 60
Error: worker queue option is invalid: worker concurrency must be between 1 and 120

shri-astro added app/astro-cli from_qa Issues Reported by QA or Automation minor team/cp-applications labels Jun 15, 2023

shri-astro assigned kushalmalani Jun 15, 2023

kushalmalani mentioned this issue Jun 22, 2023

Fix worker concurrency #1275

Merged

8 tasks

kushalmalani assigned shri-astro Jun 22, 2023

kushalmalani mentioned this issue Jun 23, 2023

Fix concurrency check for hosted deployments #1279

Merged

8 tasks

shri-astro removed their assignment Jun 26, 2023

kushalmalani mentioned this issue Jun 27, 2023

Fix deployment status #1284

Merged

8 tasks

This was referenced Jun 27, 2023

Update error message #1285

Merged

Fix zero worker concurrency error #1286

Merged

shri-astro closed this as completed Jun 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating worker concurrency doesn't reflect in the response or UI #1267

Updating worker concurrency doesn't reflect in the response or UI #1267

shri-astro commented Jun 15, 2023

kushalmalani commented Jun 22, 2023

shri-astro commented Jun 22, 2023

kushalmalani commented Jun 23, 2023

shri-astro commented Jun 26, 2023

kushalmalani commented Jun 27, 2023

shri-astro commented Jun 27, 2023

shri-astro commented Jun 27, 2023

shri-astro commented Jun 28, 2023

Updating worker concurrency doesn't reflect in the response or UI #1267

Updating worker concurrency doesn't reflect in the response or UI #1267

Comments

shri-astro commented Jun 15, 2023

kushalmalani commented Jun 22, 2023

shri-astro commented Jun 22, 2023

kushalmalani commented Jun 23, 2023

shri-astro commented Jun 26, 2023

kushalmalani commented Jun 27, 2023

shri-astro commented Jun 27, 2023

shri-astro commented Jun 27, 2023

shri-astro commented Jun 28, 2023