@@ -50,7 +50,7 @@ OpenAPI definitions, converters and LLM function calling application composer.
50
50
>
51
51
> Demonstration video composing A.I. chatbot with ` @samchon/openapi ` and [ ` agentica ` ] ( https://github.com/wrtnlabs/agentica )
52
52
>
53
- > - Shopping A.I. Chatbot Application: [ https://nestia.io/chat/shopping ] ( /chat/shopping )
53
+ > - Shopping A.I. Chatbot Application: https://nestia.io/chat/shopping
54
54
> - Shopping Backend Repository: https://github.com/samchon/shopping-backend
55
55
> - Shopping Swagger Document (` @nestia/editor ` ): [ https://nestia.io/editor/?url= ...] ( https://nestia.io/editor/?simulate=true&e2e=true&url=https%3A%2F%2Fraw.github.com%2Fsamchon%2Fshopping-backend%2Frefs%2Fheads%2Fmaster%2Fpackages%2Fapi%2Fswagger.json )
56
56
@@ -377,6 +377,75 @@ const main = async (): Promise<void> => {
377
377
main().catch(console.error);
378
378
` ` `
379
379
380
+ ### Validation Feedback
381
+ ` ` ` typescript
382
+ import { IHttpLlmFunction, IValidation } from "@samchon/openapi";
383
+ import { FunctionCall } from "pseudo";
384
+
385
+ export const correctFunctionCall = (p: {
386
+ call: FunctionCall;
387
+ functions: Array<IHttpLlmFunction<"chatgpt">>;
388
+ retry: (reason: string, errors?: IValidation.IError[]) => Promise<unknown>;
389
+ }): Promise<unknown> => {
390
+ // FIND FUNCTION
391
+ const func: IHttpLlmFunction<"chatgpt"> | undefined =
392
+ p.functions.find((f) => f.name === p.call.name);
393
+ if (func === undefined) {
394
+ // never happened in my experience
395
+ return p.retry(
396
+ "Unable to find the matched function name. Try it again.",
397
+ );
398
+ }
399
+
400
+ // VALIDATE
401
+ const result: IValidation<unknown> = func.validate(p.call.arguments);
402
+ if (result.success === false) {
403
+ // 1st trial: 30% (gpt-4o-mini in shopping mall chatbot)
404
+ // 2nd trial with validation feedback: 99%
405
+ // 3nd trial with validation feedback again: never have failed
406
+ return p.retry(
407
+ "Type errors are detected. Correct it through validation errors",
408
+ {
409
+ errors: result.errors,
410
+ },
411
+ );
412
+ }
413
+ return result.data;
414
+ }
415
+ ` ` `
416
+
417
+ Is LLM Function Calling perfect ? No , absolutely not.
418
+
419
+ LLM (Large Language Model ) service vendor like OpenAI takes a lot of type level mistakes when composing the arguments of function calling or structured output . Even though target schema is super simple like ` Array<string> ` type , LLM often fills it just by a ` string ` typed value.
420
+
421
+ In my experience , OpenAI ` gpt-4o-mini ` (` 8b ` parameters ) is taking about 70 % of type level mistakes when filling the arguments of function calling to Shopping Mall service . To overcome the imperfection of such LLM function calling , ` @samchon/openapi ` supports validation feedback strategy.
422
+
423
+ The key concept of validation feedback strategy is , let LLM function calling to construct invalid typed arguments first , and informing detailed type errors to the LLM , so that induce LLM to emend the wrong typed arguments at the next turn by using ` IHttpLlmFunction<Model>.validate() ` function.
424
+
425
+ Embedded validator function in ` IHttpLlmFunction<Model>.validate() ` is exactly same with [` typia.validate<T>() ` ](https : // typia.io/docs/validators/validate) function, so that detailed and accurate than any other validators like below. By such validation feedback strategy, 30% success rate of the 1st function calling trial has been increased to 99% success rate of the 2nd function calling trial. And have never failed from the 3rd trial.
426
+
427
+ Components | ` typia ` | ` TypeBox ` | ` ajv ` | ` io-ts ` | ` zod ` | ` C.V. `
428
+ -------------------------| --------| -----------| -------| ---------| -------| ------------------
429
+ **Easy to use ** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
430
+ [Object (simple )](https : // github.com/samchon/typia/blob/master/test/src/structures/ObjectSimple.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
431
+ [Object (hierarchical )](https : // github.com/samchon/typia/blob/master/test/src/structures/ObjectHierarchical.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
432
+ [Object (recursive )](https : // github.com/samchon/typia/blob/master/test/src/structures/ObjectRecursive.ts) | ✔ | ❌ | ✔ | ✔ | ✔ | ✔ | ✔
433
+ [Object (union , implicit )](https : // github.com/samchon/typia/blob/master/test/src/structures/ObjectUnionImplicit.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
434
+ [Object (union , explicit )](https : // github.com/samchon/typia/blob/master/test/src/structures/ObjectUnionExplicit.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ❌
435
+ [Object (additional tags )](https : // github.com/samchon/typia/#comment-tags) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
436
+ [Object (template literal types )](https : // github.com/samchon/typia/blob/master/test/src/structures/TemplateUnion.ts) | ✔ | ✔ | ✔ | ❌ | ❌ | ❌
437
+ [Object (dynamic properties )](https : // github.com/samchon/typia/blob/master/test/src/structures/DynamicTemplate.ts) | ✔ | ✔ | ✔ | ❌ | ❌ | ❌
438
+ [Array (rest tuple )](https : // github.com/samchon/typia/blob/master/test/src/structures/TupleRestAtomic.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
439
+ [Array (hierarchical )](https : // github.com/samchon/typia/blob/master/test/src/structures/ArrayHierarchical.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔
440
+ [Array (recursive )](https : // github.com/samchon/typia/blob/master/test/src/structures/ArrayRecursive.ts) | ✔ | ✔ | ✔ | ✔ | ✔ | ❌
441
+ [Array (recursive , union )](https : // github.com/samchon/typia/blob/master/test/src/structures/ArrayRecursiveUnionExplicit.ts) | ✔ | ✔ | ❌ | ✔ | ✔ | ❌
442
+ [Array (R +U , implicit )](https : // github.com/samchon/typia/blob/master/test/src/structures/ArrayRecursiveUnionImplicit.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
443
+ [Array (repeated )](https : // github.com/samchon/typia/blob/master/test/src/structures/ArrayRepeatedNullable.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
444
+ [Array (repeated , union )](https : // github.com/samchon/typia/blob/master/test/src/structures/ArrayRepeatedUnionWithTuple.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
445
+ [**Ultimate Union Type **](https : // github.com/samchon/typia/blob/master/test/src/structures/UltimateUnion.ts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌
446
+
447
+ > ` C.V. ` means ` class-validator `
448
+
380
449
### Separation
381
450
Arguments from both Human and LLM sides .
382
451
0 commit comments