Testing conventions

• package.yaml:

  ...
  
  tests:
    Test4-test:
      main:                Spec.hs
      source-dirs:         test
      ghc-options:
      - -threaded
      - -rtsopts
      - -with-rtsopts=-N
      dependencies:
      - Test4
      - tasty
      - tasty-quickcheck
      - tasty-hunit

Folder: 'project/test'

In folder test is a haskell file as root for all tests name spec.hs.

• spec.hs:

  import qualified Test.Tasty as T
  
  import qualified SpecModuleA -- test module for module ModuleA
  import qualified SpecModuleB -- test module for module ModuleB
  import qualified SpecModuleC -- test module for module ModuleC
  
  main :: IO ()
  main = 
      T.defaultMain tests
  
  tests :: T.TestTree
  tests = 
      T.testGroup 
          "Tests" 
          [
              SpecModuleA.testGroup, 
              SpecModuleB.testGroup, 
              SpecModuleC.testGroup
          ]

• example spec<Module>.hs:

  module SpecCode
      (
          testGroup
      ) where
  
  import qualified Test.Tasty as T
  import qualified Test.Tasty.QuickCheck as QC
  import qualified Test.Tasty.HUnit as HU
  
  import qualified ModuleA as A
  
  -- ...

• example spec<Module>.hs:

  -- ...
  
  testGroup :: T.TestTree
  testGroup =
      T.testGroup
          "module A"
          [
              tgClassA,
              tgUnitf1,
              tgUnitf2,
              tgUnitf3
          ]
  
  -- ...
  
  tgClassA :: T.TestTree
  tgClassA =
      T.testGroup
          "class A"
          [
              -- laws:
              tgLawA,
              tgLawB,
              tgLawC,
              -- unit tests:
              tgUnitf3,
              tgUnitlcdConvert
          ]
          
  -- ...

• Function: HUnit.testCase
  1. parameter :: [Char]/String/TestName: The test name, ends with “#”, “u” for unit test and a consecutive number. Optionally the start with a name to be distinct from other sections.
     • e.g. “Law A #u1”, “Law A #u2”, … “Law A #u33”
  2. parameter :: Assertation: Two functions have to have the same result left and right from operator “@?=”.
     • e.g. 3 * 2 @?= 2 * 3

• example spec<Module>.hs:

  -- ...
  
  tgLawA :: T.TestTree
  tgLawA =
      T.testGroup
          "Law: A"
          [
              -- ...,
              HU.testCase "#u1" (A.f4 3 HU.@?= (A.f5 3 * A.f5 3)), 
              HU.testCase "#u2" (A.f4 4 HU.@?= (A.f5 4 * A.f5 4)) -- ...
          ]
  
  -- ...

• Function: HUnit.testProperty
  1. parameter :: [Char]/String/TestName: The test name, ends with “#”, “p” for property test and a consecutive number. Optionally the start with a name to be distinct from other sections.
     • e.g. “Type 1 #p1”, “Type 1 #p2”, … “Type 1 #p42”
  2. parameter :: a: Two functions from a lambda parameter left and rigth from “=⇒”. Whereas the left on is the precondition to evaluate the rigth on, which should be an invariant beeing true.
     • e.g. 3 * 2 @?= 2 * 3

• example spec<Module>.hs:

  -- ...
  
  tgLawA :: T.TestTree
  tgLawA =
      T.testGroup
          "Law: A"
          [
              -- ...,
              QC.testProperty "#p1" (\nj -> A.f3 nj QC.==> (A.f4 nj == (A.f5 nj * A.f5 nj)) ),
              QC.testProperty "#p2" (\nj -> not A.f3 nj QC.==> (A.f4 nj == -1 ) -- ...
          ]
  
  -- ...

QuickCheck tests with test data of random samples, and the frequency distribution of the ramdom samples can be controlled by the class Arbitrary.

• Example: Let suppose there is a type that is to huge to test all possible combinations.

• type declaration of example:

  data TooHugeToTestAll = ToHuge { ra :: Int, rb :: Int, rc :: Int, rd :: Int }
      deriving (Show)

• If you want to have more tests in a certain range of values plus some in the vast possibilities then you create an orphan instance of 'Arbitrary'.

• class instance of example:

  instance QC.Arbitrary TooHugeToTestAll where
      arbitrary = ToHuge <$> f <*> f <*> f <*> f
          where
              f = QC.frequency
                  [
                      (1, QC.choose (niMin, -1)),          -- -1 or less
                      (5, QC.choose (0, niMax)),          -- 0 or more
                      (3, QC.choose (niMaxPlus1, 1000)),  -- up to 1000
                      (1, QC.choose (1001, maxBound :: Int))              -- up to maxBound
                  ]
              niMax :: Int
              niMax = 3
              niMaxPlus1 :: Int
              niMaxPlus1 = niMax + 1
              niMin :: Int
              niMin = - niMax

Background: The following instances already exist with a standard distribution for sample data.

• Arbitrary Bool
• Arbitrary Char
• Arbitrary Double
• Arbitrary Float
• Arbitrary Int
• Arbitrary Int8
• Arbitrary Int16
• Arbitrary Int32
• Arbitrary Int64
• Arbitrary Integer
• Arbitrary Ordering
• Arbitrary Word
• Arbitrary Word8
• Arbitrary Word16
• Arbitrary Word32
• Arbitrary Word64
• Arbitrary ()
• and much more … see all of them here Test.Tasty - class Arbitrary

Problem: So, how to control them differently?

Solution: By declaration of a new type (by 'newtype').

• example:

  newtype Char' = Char' Char
      deriving Show
  
  fromChar' :: Char' -> Char
  fromChar' (Char' ch) = ch
  
  instance QC.Arbitrary Char' where
      arbitrary =
          do
              genChar <- QC.frequency
                  [
                      (1, QC.elements ['A'..'Z']),
                      (1, QC.elements ['a'..'z']),
                      (1, QC.elements ['0'..'9']),
                      (5, QC.chooseAny )
                  ]
              return (Char' genChar)
  
      -- ...
  
          QC.testProperty "Char #p1" (\ch' -> (fromChar' (ch' :: Char') == '\0') QC.==> (fXYZ (fromChar' ch')) ),

• example, with a more generalised approach:

  {-# LANGUAGE FlexibleInstances #-}
  
  -- ...
  
  newtype T' a = T' a
      deriving Show
  
  fromT' :: T' a -> a
  fromT' (T' x) = x
  
  instance QC.Arbitrary (T' Char) where
      arbitrary =
          do
              genChar <- QC.frequency
                  [
                      (1, QC.elements ['A'..'Z']),
                      (1, QC.elements ['a'..'z']),
                      (1, QC.elements ['0'..'9']),
                      (5, QC.chooseAny )
                  ]
              return (T' genChar)
  
      -- ...
  
              QC.testProperty "Char #p1" (\cd' -> (fromT' (cd' :: (TH.T' Char)) == '\0') QC.==> (fXYZ (fromT' cd') == '\0') ),

The following symbols are possible for each validation criteria:

• ❓
  • not yet clear
• ❌
  • not yet
• N/A
  • not applicable
• ✅
  • criteria is met
• ✅ with comment
  • criteria is met with comment and mitigation
  • example:

    NOTE: correct functioning depends on "Cd.cdFromChar :: Char -> CharUtf8" test
    mitigation: ✅ supplemental test data (approx. 100 characters) are use in unit tests

Each function has its own test group designated as tgUnit<functionName>.

And each law has its own test group designated as tgLaw<LawName>.

Tests are validated by checking the following criteria:

• completeness:
  • the complete set of possible test values/data is used in unit tests,
  • or values for all possible equivalence classes are used
  • or random sample values/data from complete set of possible values/data is used in case of high-volume cases (QuickCheck)
• independence:
  • test methods are independent functions within the same library,
  • unless they depend on independent tested functions (if dependencies are not cyclic)
• edge cases:
  • edge cases are tested by unit tests (HUnit)
• conform doc.:
  • all tests are conform to documentation, e.g. source code comments by haddock)

After checking the above mentioned criteria, the result is documented as source code comment:

• {-  * validated: ✅
          * completeness: ✅
          * independence: ✅
          * edge cases  : ✅
          * conform doc.: ✅ -}
  tgUnitf1 :: T.TestTree
  tgUnitf1 =
      T.testGroup
          "f1"
          [
              -- ...
          ]

Each function has its own test group designated as tgClass<ClassName>.

Tests are validated by checking the following criteria:

• documented:
  • laws:
    • laws and its criteria are described and are verifyable
  • characteristics:
    • the general characteristics of the class are described
  • prefix
    • the prefix of the class is given
  • short description
    • a short description is given
• completeness:
  • the class is tested to fulfill all laws
  • the class is tested for all types and type combinations, respectively
  • and the class is tested for all types and type combinations
• stubbed:
  • if any, the default functions of the class are also tested by stub instance(s), as well

After checking the abovementioned criteria, the result is documented as source code comment:

• {-  * validated: ✅
          * documented: ✅
            * laws             : ✅
            * characteristics  : ✅
            * prefix           : ✅
            * short description: ✅
          * completeness: ✅
              * all laws documented          : ✅
              * all type combinations        : ✅
              * stubbed for default functions: ✅ -}
  tgClassA :: T.TestTree
  tgClassA =
      T.testGroup
          "class A"
          [
              -- laws:
              tgLawA,
              tgLawB,
              tgLawC,
              -- unit tests:
              tgUnitf3,
              tgUnitlcdConvert
          ]

Each module exports a test group designated as testGroup.

Tests are validated by checking the following criteria:

• documented:
  • there is completely and correct filled header like:

    Description : provides a class for ...
    Copyright   : (c) <Author>, <Year>
    License     : <License>
    Maintainer  : <E-Mail-Address>
    Stability   : experimental
    Portability : POSIX

  • It follows a general description of the purpose of the module.
• completeness:
  • all classes are in the root test group
  • all global functions have tests in the root test group
  • all classes are validated
  • all global functions are validated

• example:

  {-  * validated: ✅
          * documented: ✅
          * completeness: ✅ -}
  testGroup :: T.TestTree
  testGroup =
      T.testGroup
          "module A"
          [
              tgClassA,
              tgUnitf1,
              tgUnitf2,
              tgUnitf3
          ]