Skip to content

Commit 6effe9f

Browse files
committed
Add validation feedback strategy document
1 parent 6fedcff commit 6effe9f

File tree

1 file changed

+70
-1
lines changed

1 file changed

+70
-1
lines changed

README.md

+70-1
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ OpenAPI definitions, converters and LLM function calling application composer.
5050
>
5151
> Demonstration video composing A.I. chatbot with `@samchon/openapi` and [`agentica`](https://github.com/wrtnlabs/agentica)
5252
>
53-
> - Shopping A.I. Chatbot Application: [https://nestia.io/chat/shopping](/chat/shopping)
53+
> - Shopping A.I. Chatbot Application: https://nestia.io/chat/shopping
5454
> - Shopping Backend Repository: https://github.com/samchon/shopping-backend
5555
> - Shopping Swagger Document (`@nestia/editor`): [https://nestia.io/editor/?url=...](https://nestia.io/editor/?simulate=true&e2e=true&url=https%3A%2F%2Fraw.github.com%2Fsamchon%2Fshopping-backend%2Frefs%2Fheads%2Fmaster%2Fpackages%2Fapi%2Fswagger.json)
5656
@@ -377,6 +377,75 @@ const main = async (): Promise<void> => {
377377
main().catch(console.error);
378378
```
379379
380+
### Validation Feedback
381+
```typescript
382+
import { IHttpLlmFunction, IValidation } from "@samchon/openapi";
383+
import { FunctionCall } from "pseudo";
384+
385+
export const correctFunctionCall = (p: {
386+
call: FunctionCall;
387+
functions: Array<IHttpLlmFunction<"chatgpt">>;
388+
retry: (reason: string, errors?: IValidation.IError[]) => Promise<unknown>;
389+
}): Promise<unknown> => {
390+
// FIND FUNCTION
391+
const func: IHttpLlmFunction<"chatgpt"> | undefined =
392+
p.functions.find((f) => f.name === p.call.name);
393+
if (func === undefined) {
394+
// never happened in my experience
395+
return p.retry(
396+
"Unable to find the matched function name. Try it again.",
397+
);
398+
}
399+
400+
// VALIDATE
401+
const result: IValidation<unknown> = func.validate(p.call.arguments);
402+
if (result.success === false) {
403+
// 1st trial: 30% (gpt-4o-mini in shopping mall chatbot)
404+
// 2nd trial with validation feedback: 99%
405+
// 3nd trial with validation feedback again: never have failed
406+
return p.retry(
407+
"Type errors are detected. Correct it through validation errors",
408+
{
409+
errors: result.errors,
410+
},
411+
);
412+
}
413+
return result.data;
414+
}
415+
```
416+
417+
Is LLM Function Calling perfect? No, absolutely not.
418+
419+
LLM (Large Language Model) service vendor like OpenAI takes a lot of type level mistakes when composing the arguments of function calling or structured output. Even though target schema is super simple like `Array<string>` type, LLM often fills it just by a `string` typed value.
420+
421+
In my experience, OpenAI `gpt-4o-mini` (`8b` parameters) is taking about 70% of type level mistakes when filling the arguments of function calling to Shopping Mall service. To overcome the imperfection of such LLM function calling, `@samchon/openapi` supports validation feedback strategy.
422+
423+
The key concept of validation feedback strategy is, let LLM function calling to construct invalid typed arguments first, and informing detailed type errors to the LLM, so that induce LLM to emend the wrong typed arguments at the next turn by using `IHttpLlmFunction<Model>.validate()` function.
424+
425+
Embedded validator function in `IHttpLlmFunction<Model>.validate()` is exactly same with [`typia.validate<T>()`](https://typia.io/docs/validators/validate) function, so that detailed and accurate than any other validators like below. By such validation feedback strategy, 30% success rate of the 1st function calling trial has been increased to 99% success rate of the 2nd function calling trial. And have never failed from the 3rd trial.
426+
427+
Components | `typia` | `TypeBox` | `ajv` | `io-ts` | `zod` | `C.V.`
428+
-------------------------|--------|-----------|-------|---------|-------|------------------
429+
**Easy to use** ||||||
430+
[Object (simple)](https://github.com/samchon/typia/blob/master/test/src/structures/ObjectSimple.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
431+
[Object (hierarchical)](https://github.com/samchon/typia/blob/master/test/src/structures/ObjectHierarchical.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
432+
[Object (recursive)](https://github.com/samchon/typia/blob/master/test/src/structures/ObjectRecursive.ts) | ✔ | ❌ | ✔ | ✔ | ✔ | ✔ | ✔
433+
[Object (union, implicit)](https://github.com/samchon/typia/blob/master/test/src/structures/ObjectUnionImplicit.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
434+
[Object (union, explicit)](https://github.com/samchon/typia/blob/master/test/src/structures/ObjectUnionExplicit.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ❌
435+
[Object (additional tags)](https://github.com/samchon/typia/#comment-tags) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
436+
[Object (template literal types)](https://github.com/samchon/typia/blob/master/test/src/structures/TemplateUnion.ts) | ✔ | ✔ | ✔ | ❌ | ❌ | ❌
437+
[Object (dynamic properties)](https://github.com/samchon/typia/blob/master/test/src/structures/DynamicTemplate.ts) | ✔ | ✔ | ✔ | ❌ | ❌ | ❌
438+
[Array (rest tuple)](https://github.com/samchon/typia/blob/master/test/src/structures/TupleRestAtomic.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
439+
[Array (hierarchical)](https://github.com/samchon/typia/blob/master/test/src/structures/ArrayHierarchical.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
440+
[Array (recursive)](https://github.com/samchon/typia/blob/master/test/src/structures/ArrayRecursive.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ❌
441+
[Array (recursive, union)](https://github.com/samchon/typia/blob/master/test/src/structures/ArrayRecursiveUnionExplicit.ts) | ✔ | ✔ | ❌ | ✔ | ✔ | ❌
442+
[Array (R+U, implicit)](https://github.com/samchon/typia/blob/master/test/src/structures/ArrayRecursiveUnionImplicit.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
443+
[Array (repeated)](https://github.com/samchon/typia/blob/master/test/src/structures/ArrayRepeatedNullable.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
444+
[Array (repeated, union)](https://github.com/samchon/typia/blob/master/test/src/structures/ArrayRepeatedUnionWithTuple.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
445+
[**Ultimate Union Type**](https://github.com/samchon/typia/blob/master/test/src/structures/UltimateUnion.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
446+
447+
> `C.V.` means `class-validator`
448+
380449
### Separation
381450
Arguments from both Human and LLM sides.
382451

0 commit comments

Comments
 (0)