Merge pull request #320 from lsylvestre/master

support for test code reuse

docs/howto-write-exercises.md

@@ -32,37 +32,42 @@ to get the files for step 1, and replace `step-1` by `step-2` to
 get the files for the second step, and so on and so forth.
 
 ## The tutorials
-[Step 0 : Preliminaries](../tutorials/step-0)
+[Step 0 : Preliminaries](tutorials/step-0.md)
 
 - Structure of an exercise
 
 - Purpose of each file
 
-[Step 1: Create a trivial exercise](../tutorials/step-1)
+[Step 1: Create a trivial exercise](tutorials/step-1.md)
 
-[Step 2: Basic grading by comparison with your solution](../tutorials/step-2)
+[Step 2: Basic grading by comparison with your solution](tutorials/step-2.md)
 
 - Simple example to grade by comparison with a solution
 
 - With polymorphic functions
 
 - With multiple arguments functions
 
-[Step 3: Grading with generators for Ocaml built-in types](../tutorials/step-3)
+[Step 3: Grading with generators for Ocaml built-in types](tutorials/step-3.md)
 
 - Generate tests by using the pre-construct samplers 
 
 - Generate tests by defining its own sampler 
 
-[Step 4: Grading with generators for user-defined types](../tutorials/step-4)
+[Step 4: Grading with generators for user-defined types](tutorials/step-4.md)
 
 - Generate tests for non-parametric user-defined types 
 
 - Generate tests for parametric user-defined types 
 
-[Step 5 : More test functions](../tutorials/step-5)
+[Step 5 : More test functions](tutorials/step-5.md)
 
-[Step 6 : Grading functions for variables](../tutorials/step-6)
-
-[Step 7 : Introspection of students code](../tutorials/step-7)
+[Step 6 : Grading functions for variables](tutorials/step-6.md)
 
+[Step 7 : Modifying the comparison functions (testers) with the optional arguments [~test], [~test_stdout], [~test_stderr]](tutorials/step-7.md)
+
+[Step 8 : Reusing the grader code](tutorials/step-8.md)
+
+- Separating the grader code
+
+[Step 9 : Introspection of students code](tutorials/step-9.md)    

docs/tutorials/step-8.md

@@ -0,0 +1,190 @@
+# Step 8: Reusing the grader code
+
+This step explains how to separate the grader code, and eventually reuse it in
+other exercises. 
+
+During the grading, the file **test.ml** is evaluated in an environment that
+contains notably: 
+- **prelude.ml** and **prepare.ml** ;
+- the student code isolated in a module `Code` ;
+- **solution.ml** in a module `Solution` ;
+- the grading modules **Introspection**, **Report** and **Test_lib**.
+
+### Separating the grader code
+
+It is possible to extend this environment by declaring some other user-defined
+modules in an optional file **depend.txt**, located in the exercise directory.
+
+Each declaration in **depend.txt** is a single line containing the relative path
+of an *.ml* or *.mli* file. The order of the *.ml* declarations specifies the
+order in which each module is loaded in the grading environment.
+
+By default each dependency *foo.ml* is isolated in a module *Foo*, which can be
+constrained by the content of an optional signature file *foo.mli*. Furthermore,
+an annotation `[@@@included]` can be used at the beginning of a file *foo.ml* to
+denote that all the bindings of *foo.ml* are evaluated in the toplevel
+environment (and not in a module *Foo*). 
+
+Dependencies that are not defined at the root of the exercise repository are
+ignored by the build system: therefore, if you modify them, do not forget to
+refresh the timestamp of `test.ml` (using `touch` for instance).
+
+### A complete example
+
+Let's write an exercise dedicated to *Peano numbers*. Here is the structure of
+the exercise:
+
+```
+.
+├── exercises
+│   ├── index.json
+│   └── lib
+│   │   ├── check.ml
+│   │   └── check.mli
+│   ├── peano
+│   │   ├── depend.txt
+│   │   ├── descr.md
+│   │   ├── meta.json
+│   │   ├── prelude.ml
+│   │   ├── prepare.ml
+│   │   ├── solution.ml
+│   │   ├── template.ml
+│   │   ├── test.ml
+│   │   └── tests
+│   │       ├── samples.ml
+│   │       ├── add.ml
+│   │       └── odd_even.ml
+│   ├── an-other-exercise
+│   │   ├── depend.txt
+│   │   │ ...
+```
+The exercise **peano** follows the classical format : **prelude.ml**,
+**prepare.ml**, **solution.ml**, **template.ml** and **test.ml**.
+It also includes several dependencies (**check.ml**, **samples.ml**, **add.ml**
+and **odd_even.ml**) which are declared as follows in **depend.txt**:
+
+```txt
+../lib/check.mli
+../lib/check.ml    # a comment
+
+tests/samples.ml
+tests/add.ml
+tests/odd_even.ml
+``` 
+
+Here is in details the source code of the exercise :
+
+- **descr.md**
+
+> * implement the function `add : peano -> peano -> peano` ; 
+> * implement the functions `odd : peano -> bool` and `even : peano -> bool`.
+
+- **prelude.ml**
+```ocaml
+type peano = Z | S of peano
+```
+
+- **solution.ml**
+```ocaml
+let rec add n = function
+| Z -> n
+| S m -> S (add n m)
+
+let rec odd = function
+| Z -> false 
+| S n -> even n
+and even = function
+| Z -> true
+| S n -> odd n
+```
+
+- **test.ml**
+```ocaml
+let () =
+Check.safe_set_result [ Add.test ; Odd_even.test ]
+```
+
+Note that **test.ml** is very compact because it simply combines functions
+defined in separated files.
+
+- **../lib/check.ml**:
+```ocaml
+open Test_lib
+open Report
+
+let safe_set_result tests =
+set_result @@
+ast_sanity_check code_ast @@ fun () ->
+List.mapi (fun i test -> 
+Section ([ Text ("Question " ^ string_of_int i ^ ":") ],
+test ())) tests
+```
+- **../lib/check.mli**:
+```ocaml
+val safe_set_result : (unit -> Report.t) list -> unit
+```
+
+- **tests/add.ml**: 
+```ocaml
+let test () =
+Test_lib.test_function_2_against_solution
+[%ty : peano -> peano -> peano ] "add"
+[ (Z, Z) ; (S(Z), S(S(Z))) ]
+```
+- **tests/odd_even.ml** :
+```ocaml
+let test () =
+Test_lib.test_function_1_against_solution
+[%ty : peano -> bool ] "odd"
+[ Z ; S(Z) ; S(S(Z)) ] 
+@
+Test_lib.test_function_1_against_solution
+[%ty : peano -> bool ] "even"
+[ Z ; S(Z) ; S(S(Z)) ]
+```
+Remember that **Test_lib** internally requires a user-defined sampler
+`sample_peano : unit -> peano` to generate value of type `peano`. This sampler
+has to be present in the toplevel environment -- and not in a module -- in order
+to be found by the introspection primitives during grading. Therefore,
+we define this sampler in a file starting with the annotation `[@@@included]`.
+- **tests/samples.ml**:
+```ocaml
+[@@@included]
+
+let sample_peano () =
+let rec aux = function
+| 0 -> Z
+| n -> S (aux (n-1)) 
+in aux (Random.int 42)
+```
+
+Finally, the content of **test.ml** will be evaluated in the following
+environment:
+
+```ocaml
+val print_html : 'a -> 'b
+type peano = Z | S of peano
+module Code :
+sig
+val add : peano -> peano -> peano
+val odd : peano -> bool
+val even : peano -> bool
+end
+module Solution :
+sig
+val add : peano -> peano -> peano
+val odd : peano -> bool
+val even : peano -> bool
+end
+module Test_lib : Test_lib.S
+module Report = Learnocaml_report
+module Check : sig val check_all : (unit -> Report.t) list -> unit end
+val sample_peano : unit -> peano
+module Add : sig val test : unit -> Report.t end
+module Odd_even : sig val test : unit -> Report.t end
+```
+
+In the end, this feature can provide an increased comfort for writing large a
+utomated graders and for reusing them in other exercises.
+
+

src/grader/grading.ml

@@ -142,6 +142,50 @@ let get_grade
       Toploop_ext.use_string ~print_outcome ~ppf_answer
         "module Report = Learnocaml_report" ;
       set_progress [%i"Launching the test bench."] ;
+
+      let () =
+        let open Learnocaml_exercise in
+        let files = File.dependencies (access File.depend exo) in
+        let rec load_dependencies signatures = function
+        | [] -> () (* signatures without implementation are ignored *)
+        | file::fs ->
+          let path = File.key file
+          and content = decipher file exo in
+          let modname = String.capitalize_ascii @@
+                        Filename.remove_extension @@ Filename.basename path in
+          match Filename.extension path with
+          | ".mli" -> load_dependencies ((modname,content) :: signatures) fs
+          | ".ml" -> 
+            let included,content = 
+              (* the first line of an .ml file can contain an annotation       *)
+              (* [@@@included] which denotes that this file has to be included *)
+              (* directly in the toplevel environment, and not in an module.   *)
+              match String.index_opt content '\n' with
+              | None -> (false,content)
+              | Some i -> 
+                (match String.trim (String.sub content 0 i) with 
+                 | "[@@@included]" -> 
+                    let content' = String.sub content i @@ 
+                                   (String.length content - i)
+                    in (true,content')
+                 | _ -> (false,content))
+            in
+            (handle_error (internal_error [%i"while loading user dependencies"]) @@
+             match included with
+             | true -> Toploop_ext.use_string ~print_outcome ~ppf_answer 
+                                  ~filename:(Filename.basename path) content 
+             | false ->
+               let use_mod = 
+                 Toploop_ext.use_mod_string ~print_outcome ~ppf_answer ~modname in
+               match List.assoc_opt modname signatures with 
+               | Some sig_code -> use_mod ~sig_code content
+               | None -> use_mod content); 
+               load_dependencies signatures fs
+          | _ -> failwith ("uninterpreted dependency \"" ^ path ^
+                           "\", file extension expected : .ml or .mli") in 
+          load_dependencies [] files
+      in
+
       handle_error (internal_error [%i"while testing your solution"]) @@
       Toploop_ext.use_string ~print_outcome ~ppf_answer ~filename:(file "test.ml")
         (Learnocaml_exercise.(decipher File.test exo)) ;