|
|
@@ -0,0 +1,139 @@
|
|
|
+How to merge the testsuite into your codebase
|
|
|
+=============================================
|
|
|
+
|
|
|
+- Copy the entire "test" directory into the root directory of the framework.
|
|
|
+ The framework already contains a test directory with files test1.cvc and
|
|
|
+ test2.cvc, you can remove these.
|
|
|
+
|
|
|
+
|
|
|
+- Add the following lines to your Makefile:
|
|
|
+
|
|
|
+###############################################################################
|
|
|
+#
|
|
|
+# Testing
|
|
|
+#
|
|
|
+
|
|
|
+check: all
|
|
|
+ @cd test; \
|
|
|
+ CIVAS=../$(TEST_CIVAS) \
|
|
|
+ CIVVM=../$(TEST_CIVVM) \
|
|
|
+ CIVCC=../$(TEST_CIVCC) \
|
|
|
+ RUN_FUNCTIONAL=$(TEST_RUN_FUNCTIONAL) \
|
|
|
+ bash run.bash $(TEST_DIRS)
|
|
|
+
|
|
|
+
|
|
|
+- Add the following lines to your Makefile.Targets (this is configuration, so
|
|
|
+ edit them accordingly):
|
|
|
+
|
|
|
+###############################################################################
|
|
|
+#
|
|
|
+# Testing
|
|
|
+#
|
|
|
+# Uncomment the necessary parts when adding extensions.
|
|
|
+
|
|
|
+# Subdirectories of "test/" to traverse into during testing. Enable
|
|
|
+# "nested_funs" and "arrays" when you start implementing those extensions.
|
|
|
+# "preprocess" is optional, for those who implemented the C preprocessor.
|
|
|
+TEST_DIRS := basic # nested_funs arrays preprocess
|
|
|
+
|
|
|
+# Flags to pass to your compiler when running tests (e.g., a verbosity level).
|
|
|
+TEST_CFLAGS := ""
|
|
|
+
|
|
|
+# Whether to run tests that compare program output. Set to 1 after you have
|
|
|
+# implemented your assembly phase.
|
|
|
+TEST_RUN_FUNCTIONAL := 0
|
|
|
+
|
|
|
+# Path to toolchain files, relative to the root directory (used for functional
|
|
|
+# test).
|
|
|
+TEST_CIVAS := bin32/civas
|
|
|
+TEST_CIVVM := bin32/civvm
|
|
|
+
|
|
|
+# Path to your compiler binary, relative to the root directory.
|
|
|
+TEST_CIVCC := bin/civicc
|
|
|
+
|
|
|
+
|
|
|
+Types of tests
|
|
|
+==============
|
|
|
+
|
|
|
+The "test" directory has a number of subdirectories that group the tests into
|
|
|
+different types:
|
|
|
+- The "basic" directory contains tests that test the minimal requirements as
|
|
|
+ explained in the milestones and reference manual, without any extensions.
|
|
|
+- The "nested_funs" and "arrays" directories contain tests that test the
|
|
|
+ functyionalities added by the corresponding extensions.
|
|
|
+- The "preprocess" directory is completely optional, for those students who
|
|
|
+ chose to implement parsing of C preprocessor directives and have a compiler
|
|
|
+ pass that calls the C preprocessor (the "cpp" command).
|
|
|
+
|
|
|
+
|
|
|
+The second nesting level of the test directory has four different formats:
|
|
|
+
|
|
|
+- check_success: Small tests that only check if your compiler exits with a 0
|
|
|
+ status code.
|
|
|
+
|
|
|
+- check_error: Small tests that only check if your compiler exits with a
|
|
|
+ non-zero status code.
|
|
|
+
|
|
|
+- functional: These are the important tests. Your compiler is used to compile a
|
|
|
+ program, which is then assembled and executed with the CiviC VM. The output
|
|
|
+ is compared to the expected output, which is in the corresponding .out file.
|
|
|
+
|
|
|
+- combined_*: This is a special type of functional test, which requires
|
|
|
+ multiple source files to be run at the same time, where one of the source
|
|
|
+ files contains a main function. This is used for testing external variables,
|
|
|
+ which requires a file that exports a variable and a file that imports that
|
|
|
+ variable.
|
|
|
+
|
|
|
+
|
|
|
+How to use the testsuite
|
|
|
+========================
|
|
|
+
|
|
|
+Run all tests by executing "make check" in a terminal. Failed tests will show
|
|
|
+the output that your compiler generated while running the test.
|
|
|
+
|
|
|
+The test suite is designed to be part of your workflow, for example:
|
|
|
+1. Implement a compiler feature (e.g., a set of grammar rules in your parser).
|
|
|
+2. Run "make check".
|
|
|
+3. For the tests that fail, see if the printed output of your compiler helps
|
|
|
+ you to debug the program.
|
|
|
+4. If the output of "make check" is insufficient, have a look at the source
|
|
|
+ files of the failed tests, add some print statements to your code, and try
|
|
|
+ to get your compiler to work with only those source files.
|
|
|
+5. When you have debugged a test case, run "make check" again to verify that
|
|
|
+ the test now succeeds and that you have not broken any other tests.
|
|
|
+
|
|
|
+
|
|
|
+On a more global level, you have to enable more tests in your Makefile.Targets
|
|
|
+file as you advance in the course. First of all, you should uncomment the
|
|
|
+"arrays" and "nested_funs" targets of TEST_DIRS when you start working on these
|
|
|
+extensions. Second, you should set TEST_FUNCTIONAL to 1 when you have
|
|
|
+implemented the first version of your code generation phase, to enable tests
|
|
|
+that compare program output.
|
|
|
+
|
|
|
+
|
|
|
+FAQ
|
|
|
+===
|
|
|
+
|
|
|
+Q: "If a lot of tests fail, will I fail the assignment?"
|
|
|
+A: Obviously, the more tests succeed, the higher your grade is likely to be.
|
|
|
+ Some tests, however, are more important than others (e.g., functional tests
|
|
|
+ are important). Some features may also (unintendedly) have more test cases
|
|
|
+ than others. For example, when your parser does not support function
|
|
|
+ definitions yet, most tests will fail since almost all are wrapped in a
|
|
|
+ "void foo" function. We will try to look closely to exactly which tests fail
|
|
|
+ when evaluating your compilers, and not only to the number of failing tests.
|
|
|
+
|
|
|
+Q: "If all test succeed, will I get a 10?"
|
|
|
+A: You will probably get close, but we also review code quality. If your C coding
|
|
|
+ style is horrible, we will subtract points, even if all of the tests
|
|
|
+ succeed.
|
|
|
+
|
|
|
+Q: "I have my own testsuite, should I switch to yours instead?"
|
|
|
+A: Using the testsuite is not mandatory. If you have a fancy automated
|
|
|
+ testsuite ow your own that covers all input cases, then you can surely use
|
|
|
+ that. Do keep in mind that we will use our own test suite to evaluate your
|
|
|
+ compilers, so it is recommended to at run the testsuite at least once to see
|
|
|
+ if you missed anything.
|
|
|
+
|
|
|
+Q: "Does the testsuite output look awesome with fancy terminal colors?"
|
|
|
+A: Yes :-)
|
Taddeus Kroes