Automated Testing and Conditional Compilation
Program PassStats requires runs with different data sets to test it. After any changes in the code we should test the program fully again. This regression testing is used to check that changes have not introduced new faults into the system. This can be tedious and the programmer is likely to use a subset of the original tests and to overlook some possible consequences of the changed code. The following program TestStats has a test procedure that repeatedly (i) instructs procedure CalcStats to process a test file, (ii) compares the results with those expected for that file and (iii) outputs the test results. The expected results for tn.csv (where n is a digit) are in file expectedn.csv. Each line in an expected file comprises either strMalePassRate,strFemalePassRate or error,n (where n is the number of the expected error message).
The code as written below creates an executable file that performs the above testing, because it has the {$DEFINE DEBUG} compiler directive. To obtain an executable to perform like Program PassStats, you must comment out this directive. Conventionally, we do this by inserting a full stop before the dollar sign. The $ no longer follows immediately after the brace, so the directive is changed into a comment. Regression testing becomes easy!
A screenshot of the output follows the code. We describe test files t8.csv and t9.csv during our discussion of white box testing below.
program TestStats1; {$APPTYPE CONSOLE} {$DEFINE DEBUG} //Insert a full stop before the $ to compile the program without the test. //Reads scores for males and females from a csv file //and outputs pass rate for each. uses SysUtils, Strutils; const PASSMARK = 40; ERROR_MESSAGES : array [0..6] of string = ('', 'File not found', 'File empty', 'Comma should follow letter', 'First letter must be M or F', 'Mark must be between 0 and 100 inclusive', 'An integer must be after the comma'); var intCurrentScore, intMales, intFemales, intMalePasses, intFemalePasses, intLine, intError, intCount : integer; strCurrentScore, strMalePassRate, strFemalepassRate : string; charCurrentGender : char; rMalePassRate, rFemalePassRate : real; ErrorFound : Boolean; procedure CalcStats(strMarksFile : string); var Marks, Results : Text; strCurrentLine : string; CommaPos, ErrorCode : integer; procedure OutPutError; begin ErrorFound := True; write(ERROR_MESSAGES[intError]); writeln(Results, 'error' + ',' + intToStr(intError)); if intLine = 0 then writeln else writeln(' at line ', intLine); end; //nested proc begin intError := 0; intLine := 0; assignFile(Results, 'results.csv'); rewrite(Results); if not fileExists(strMarksFile) then begin intError := 1; OutPutError; end else begin intMales := 0; intFemales := 0; intMalePasses := 0; intFemalepasses := 0; intCurrentScore := 0; ErrorFound := False; assignFile(Marks, strMarksFile); reset(Marks); if eof(Marks) then begin intError := 2; OutPutError; end else begin while not eof(Marks) do begin repeat readln(Marks, strCurrentLine); inc(intLine); until ((strCurrentLine <> ',') and (strCurrentLine <> '')) or eof(Marks); //Blank line or comma may be at the end of the file if not ((strCurrentLine = '') or (strcurrentline = ',')) then begin CommaPos := pos(',', strCurrentline); if (CommaPos <> 2) then begin intError := 3; //Comma should follow letter OutPutError; end else begin charCurrentGender := LeftStr(strCurrentLine, 1)[1]; charCurrentGender := UpCase(charCurrentGender); if not (charCurrentGender in ['M', 'F']) then begin intError := 4; OutPutError; end else begin strCurrentScore := rightStr(strCurrentLine, length(strCurrentLine) - 2); val(strCurrentScore, intCurrentScore, ErrorCode); if not (ErrorCode = 0) then begin intError := 6; OutPutError; end else begin if (intCurrentScore < 0) or (intCurrentScore > 100) then begin intError := 5; OutPutError; end else //no error detected begin if charCurrentGender = 'M' then begin inc(intMales); if intCurrentscore >= PASSMARK then inc(intMalePasses); end else //females begin inc(intFemales); if intCurrentscore >= PASSMARK then inc(intFemalePasses); end; end; //if (intCurrentScore < 0) or (intCurrentScore > 100) end; //if not (ErrorCode = 0) end; //if not (charCurrentGender in ['M', 'F']) end; //if CommaPos <> 2 end; //if not ((strCurrentLine = '') or (strcurrentline = ',')) end; //while if not ErrorFound then begin if intMales = 0 then begin writeln('No males'); strMalePassRate := 'NoMales'; end else begin rMalePassRate := intMalePasses * 100 / intMales; strMalePassRate := FloatToStrf(rMalePassRate, ffFixed, 6, 2); writeln('Male Pass Rate (%): ', strMalePassRate); end; if intFemales = 0 then begin writeln('No females'); strFemalePassRate := 'NoFemales'; end else begin rFemalePassRate := intFemalePasses * 100 / intFemales; strFemalePassRate := FloatToStrf(rFemalePassRate, ffFixed, 6, 2); writeln('Female Pass Rate (%): ', strFemalePassRate); end; writeln(Results, strMalePassRate + ',' + strFemalePassRate); end; //if not ErrorFound end; //if eof(Marks) end; //if not fileExists(FileName) closeFile(Results); if fileExists(strMarksFile) then closeFile(Marks); end; {$IFDEF DEBUG} procedure Test(File_No: integer); var intLineNo : integer; strExpected, strResults, strTestfile, strExpectedFile : string; ExpectedFile, ResultsFile : text; ProgError : Boolean; begin ProgError := False; strTestFile := 't' + intToStr(File_No) + '.csv'; strExpectedFile := 'expected' + intToStr(File_No) + '.csv'; writeln(#13#10'Output from test file ', File_No, ':'); CalcStats(strTestFile); if fileExists(strExpectedFile) then assignFile(ExpectedFile, strExpectedFile); if fileExists('results.csv') then assignFile(ResultsFile, 'results.csv'); reset(ExpectedFile); reset(ResultsFile); //Compare files line by line intLineNo := 0; while not eof(ExpectedFile) and not eof(ResultsFile) do begin readln(ExpectedFile, strExpected); readln(ResultsFile, strResults); inc(intLineNo); if strExpected <> strResults then begin ProgError := True; writeln(' MISMATCH at Line ', intLineNo); writeln('Expected: ',strExpected, ' Found: ', strResults); end; if eof(ExpectedFile) <> eof(ResultsFile) then begin ProgError := True; writeln(' WRONG number of lines in results file'); end; end; if ProgError = False then writeln(' No programming error detected by file ', File_No); closeFile(ExpectedFile); closeFile(ResultsFile); end; //proc {$ENDIF} begin {$IfDEF DEBUG} for intCount := 1 to 9 do Test(intCount); {$ELSE} CalcStats('marksheet.csv'); {$ENDIF} readln; end.

Output from 9 test files
The results are encouraging. Now we need to look at the code and carry out white box testing. We should identify first any lines that we have not tested. Testing line coverage manually is tedious but not difficult. Computing students will be familiar with dry runs through the code for given input data. This is made easier by the ability to step through the code (just press F7 repeatedly in Lazarus or Delphi 2007) and note the sequence of line numbers. In Lazarus, not only is the current line highlighted but also a green arrow points to the line number.
dcc32 -gd TestStats1.dprWe ran the code coverage tool with the command
CodeCoverage -m TestStats1.map -e TestStats1.exe -u TestStats1.dprand obtained this HTML file of the results.
Lines which the compiler has not needed to translate have a white background. Lines highlighted in green have been run, but those in purple have not. We have run all of the lines in the procedure that we have been testing, but not our lines to report programming errors. (We had showed that these worked as intended by introducing temporary errors to the code of procedure CalcStats), but this tutorial focuses on testing the code of procedure CalcStats).
Having seen that we have run all the lines of code we wanted to test, (and thereby tested each branch of each if statement), we need to examine whether or not we have tested each possible combination of multiple conditions in expressions. In the following analysis of line 75, the first two conditions cannot both be False (F). We show in the table only one test case for each combination.
Condition | Meaning | Test Case | ||
strCurrentLine <> ',' | strCurrentLine <> '' | eof(Marks) | ||
T | T | T | End of file neither blank nor lone comma | 17 |
T | T | F | Neither blank nor lone comma before end of file | 16 |
T | F | T | Blank line at end of file | 14 |
T | F | F | Blank line before end of file | 12 |
F | T | T | Lone comma at end of file | 22 |
F | T | F | Lone comma before end of file | 11 |
Our black box testing has done a good job of covering the possible combinations. For line 78 we have:
Condition | Test Case | |
strCurrentLine = ',' | strCurrentLine = '' | |
T | F | 11 |
F | T | 12 |
F | F | 15 |
We have tested all possible combinations of conditions. Similarly, our black box test cases for line 107 suffice:
Condition | Test Case | |
intCurrentScore < 0 | (intCurrentScore > 100 | |
T | F | 3 |
F | T | 4 |
F | F | 15 |
Byron Weber Becker provides useful Pascal templates, including testing recommendations, for common tasks. A while loop should be tested so that the loop executes "0 times, 1 time, many times. If there is a maximum, test it too." Our loop beginning while not eof(Marks) cannot be tested zero times because it is only executed if the file is not empty. We have tested it up to 100 times, but have no test files with a single line. (We overlooked this possibility in our black box testing). File t8.csv containing the data M,55 gave the expected result of male pass rate 100.00% and the message "No females".
repeat readln(Marks, strCurrentLine); inc(intLine); until ((strCurrentLine <> ',') and (strCurrentLine <> '')) or eof(Marks);We are starting before the end of the file so we do always want to read at least one line. We have analysed the combinations of conditions and shown that each has been tested at least once. We now test for many times through the loop with f9.csv, containing the lines M,40 and F,100 separated by 20 blank lines.
Our test plan with results continues as follows:
Test Case | File | Line | Data | Reason for test | Expected result | Actual Result |
24 | t8.csv | 1 | M,55 | One pass through loop | Male pass rate 100.00%, 'No females' | As expected |
25 | t9.csv | 2-21 | blank lines | Many passes through loop | Male and female pass rates 100.00% | As expected |
For procedure CalcStats, thorough black box testing includes many test cases that we would have used for white box testing. The combination of both approaches gives us some confidence that the procedure is robust.