vllm compatibility with latest release

[lynkz-matt-psaltis]

Description

Ray currently uses Pydantic v2's v1 compatibility layer which isn't compatible with newer versions of vllm which have fully migrated to v2. I'm currently working on aligning Ray and RayLLM with package versions that align to vllm and wanted to open a dialog around potentially contributing this work back to the community.

Use case

No response

[shrekris-anyscale]

Thanks for opening this @lynkz-matt-psaltis! To clarify– are you trying to migrate Ray and RayLLM to use Pydantic v2 internally?

One of the reasons that Ray (and Serve in particular) still uses the Pydantic v1 APIs is to allow users with Pydantic v1 applications to still run Serve. We have a compatibility layer that makes this possible. For the time being, we're planning to keep this structure to provide support for as many applications as possible.

Is there a reason that vllm is incompatible when Ray uses the Pydantic v1 APIs internally? I'm not sure why that would affect vllm.

[lynkz-matt-psaltis]

Hi @shrekris-anyscale thanks for reaching out. Our goal is to be able to run latest VLLM with RayLLM. RayLLM is currently pinned to a range in the vllm 0.2.x versions however 0.3.x and now 0.4.0 has dropped today.

The compatibility problems come from the interop between VLLM and Ray mixing Pydantic v1 & v2 base models.
Ray's models are Pydantic.v1 compatibility.
VLLM's Models are now Pydantic v2 native (This happened in the 0.3.0 version)

Pydantic's stance on this configuration is:

Mixing of V1 and V2 models is not supported which means that type parameters of such generic BaseModel (V2) cannot be V1 models.

See: https://docs.pydantic.dev/latest/migration/#:~:text=Mixing%20of%20V1%20and%20V2,V2)%20cannot%20be%20V1%20models.

So we set about doing an internal upgrade of both Ray and RayLLM to remove the use of the Pydantic v1 compatibility layer and move fully to v2. This works when combined with a number of code and signature changes due to breaking changes in VLLM.

It is my understanding that while Ray continues to use the Pydantic v1 compatibility layer, VLLM must remain pinned to 0.2.x versions.

Sounds like this could get interesting - Would love to know your thoughts :)

[shrekris-anyscale]

What failures do you see if you install Pydantic v2 and proceed to use Ray (with the compatibility layer) and vLLM?

The compatibility problems come from the interop between VLLM and Ray mixing Pydantic v1 & v2 base models.

This is surprising because we've had a number of users run Pydantic v2 with Ray successfully. The point of the compatibility layer is to enable that. I'm curious what issues are popping up.