====== Testing conventions ======

===== Setup with stack =====

==== package.yaml ====
  * package.yaml: <code>
...

tests:
  Test4-test:
    main:                Spec.hs
    source-dirs:         test
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - Test4
    - tasty
    - tasty-quickcheck
    - tasty-hunit
</code>

===== Test preparation =====

Folder: 'project/test'

==== Root: Spec.hs ====

In folder test is a haskell file as root for all tests name ''spec.hs''.

  * ''spec.hs'':<code>
import qualified Test.Tasty as T

import qualified SpecModuleA -- test module for module ModuleA
import qualified SpecModuleB -- test module for module ModuleB
import qualified SpecModuleC -- test module for module ModuleC

main :: IO ()
main = 
    T.defaultMain tests

tests :: T.TestTree
tests = 
    T.testGroup 
        "Tests" 
        [
            SpecModuleA.testGroup, 
            SpecModuleB.testGroup, 
            SpecModuleC.testGroup
        ]
</code>

==== Test module: Spec<Module>.hs ====

=== Beginning ===

  * example ''spec<Module>.hs'':<code>
module SpecCode
    (
        testGroup
    ) where

import qualified Test.Tasty as T
import qualified Test.Tasty.QuickCheck as QC
import qualified Test.Tasty.HUnit as HU

import qualified ModuleA as A

-- ...
</code>

=== Test tree by test group ===

  * example ''spec<Module>.hs'':<code>
-- ...

testGroup :: T.TestTree
testGroup =
    T.testGroup
        "module A"
        [
            tgClassA,
            tgUnitf1,
            tgUnitf2,
            tgUnitf3
        ]

-- ...

tgClassA :: T.TestTree
tgClassA =
    T.testGroup
        "class A"
        [
            -- laws:
            tgLawA,
            tgLawB,
            tgLawC,
            -- unit tests:
            tgUnitf3,
            tgUnitlcdConvert
        ]
        
-- ...
</code>

=== HUnit test ===

  * Function: HUnit.testCase
    - parameter :: [Char]/String/TestName: The test name, ends with "#", "u" for unit test and a consecutive number. Optionally the start with a name to be distinct from other sections.
      * e.g. "Law A #u1", "Law A #u2", ... "Law A #u33"
    - parameter :: Assertation: Two functions have to have the same result left and right from operator "@?=".
      * e.g. 3 * 2 @?= 2 * 3

  * example ''spec<Module>.hs'':<code>
-- ...

tgLawA :: T.TestTree
tgLawA =
    T.testGroup
        "Law: A"
        [
            -- ...,
            HU.testCase "#u1" (A.f4 3 HU.@?= (A.f5 3 * A.f5 3)), 
            HU.testCase "#u2" (A.f4 4 HU.@?= (A.f5 4 * A.f5 4)) -- ...
        ]

-- ...
</code>

=== QuickCheck test ===

  * Function: HUnit.testProperty
    - parameter :: [Char]/String/TestName: The test name, ends with "#", "p" for property test and a consecutive number. Optionally the start with a name to be distinct from other sections.
      * e.g. "Type 1 #p1", "Type 1 #p2", ... "Type 1 #p42"
    - parameter :: a: Two functions from a lambda parameter left and rigth from "==>". Whereas the left on is the precondition to evaluate the rigth on, which should be an invariant beeing true.
      * e.g. 3 * 2 @?= 2 * 3

  * example ''spec<Module>.hs'':<code>
-- ...

tgLawA :: T.TestTree
tgLawA =
    T.testGroup
        "Law: A"
        [
            -- ...,
            QC.testProperty "#p1" (\nj -> A.f3 nj QC.==> (A.f4 nj == (A.f5 nj * A.f5 nj)) ),
            QC.testProperty "#p2" (\nj -> not A.f3 nj QC.==> (A.f4 nj == -1 ) -- ...
        ]

-- ...
</code>

===== Test types in more detail =====

==== QuickCheck ====

=== Control of test samples ===

QuickCheck tests with test data of random samples, and the frequency distribution of the ramdom samples can be controlled by the class Arbitrary.

  * Example: Let suppose there is a type that is to huge to test all possible combinations.

    * type declaration of example: <code>
data TooHugeToTestAll = ToHuge { ra :: Int, rb :: Int, rc :: Int, rd :: Int }
    deriving (Show)
</code>

  * If you want to have more tests in a certain range of values plus some in the vast possibilities then you create an orphan instance of 'Arbitrary'.

    * class instance of example: <code>
instance QC.Arbitrary TooHugeToTestAll where
    arbitrary = ToHuge <$> f <*> f <*> f <*> f
        where
            f = QC.frequency
                [
                    (1, QC.choose (niMin, -1)),          -- -1 or less
                    (5, QC.choose (0, niMax)),          -- 0 or more
                    (3, QC.choose (niMaxPlus1, 1000)),  -- up to 1000
                    (1, QC.choose (1001, maxBound :: Int))              -- up to maxBound
                ]
            niMax :: Int
            niMax = 3
            niMaxPlus1 :: Int
            niMaxPlus1 = niMax + 1
            niMin :: Int
            niMin = - niMax
</code>

=== Control of test samples for known types ===

Background: The following instances already exist with a standard distribution for sample data.

  * Arbitrary Bool	 
  * Arbitrary Char	 
  * Arbitrary Double	 
  * Arbitrary Float	 
  * Arbitrary Int	 
  * Arbitrary Int8	 
  * Arbitrary Int16	 
  * Arbitrary Int32	 
  * Arbitrary Int64	 
  * Arbitrary Integer	 
  * Arbitrary Ordering	 
  * Arbitrary Word	 
  * Arbitrary Word8	 
  * Arbitrary Word16	 
  * Arbitrary Word32	 
  * Arbitrary Word64	 
  * Arbitrary ()
  * and much more ... see all of them here [[https://hackage.haskell.org/package/tasty-quickcheck-0.10.2/docs/Test-Tasty-QuickCheck.html#t:Arbitrary|Test.Tasty - class Arbitrary]]

Problem: So, how to control them differently?

Solution: By declaration of a new type (by 'newtype').

  * example: <code>
newtype Char' = Char' Char
    deriving Show

fromChar' :: Char' -> Char
fromChar' (Char' ch) = ch

instance QC.Arbitrary Char' where
    arbitrary =
        do
            genChar <- QC.frequency
                [
                    (1, QC.elements ['A'..'Z']),
                    (1, QC.elements ['a'..'z']),
                    (1, QC.elements ['0'..'9']),
                    (5, QC.chooseAny )
                ]
            return (Char' genChar)

    -- ...

        QC.testProperty "Char #p1" (\ch' -> (fromChar' (ch' :: Char') == '\0') QC.==> (fXYZ (fromChar' ch')) ),
</code>

  * example, with a more generalised approach: <code>
{-# LANGUAGE FlexibleInstances #-}

-- ...

newtype T' a = T' a
    deriving Show

fromT' :: T' a -> a
fromT' (T' x) = x

instance QC.Arbitrary (T' Char) where
    arbitrary =
        do
            genChar <- QC.frequency
                [
                    (1, QC.elements ['A'..'Z']),
                    (1, QC.elements ['a'..'z']),
                    (1, QC.elements ['0'..'9']),
                    (5, QC.chooseAny )
                ]
            return (T' genChar)

    -- ...

            QC.testProperty "Char #p1" (\cd' -> (fromT' (cd' :: (TH.T' Char)) == '\0') QC.==> (fXYZ (fromT' cd') == '\0') ),
</code>

===== Validation =====

The following symbols are possible for each validation criteria:
  * ❓
    * not yet clear
  * ❌
    * not yet
  * N/A
    * not applicable
  * ✅
    * criteria is met
  * ✅ with comment
    * criteria is met with comment and mitigation
    * example: <code>
NOTE: correct functioning depends on "Cd.cdFromChar :: Char -> CharUtf8" test
mitigation: ✅ supplemental test data (approx. 100 characters) are use in unit tests
</code>

==== Functions and laws ====

Each function has its own test group designated as ''tgUnit<functionName>''.

And each law has its own test group designated as ''tgLaw<LawName>''.

Tests are validated by checking the following criteria:
  * completeness: 
    * the complete set of possible test values/data is used in unit tests, 
    * or values for all possible equivalence classes are used 
    * or random sample values/data from complete set of possible values/data is used in case of high-volume cases (QuickCheck)
  * independence:
    * test methods are independent functions within the same library, 
    * unless they depend on independent tested functions (if dependencies are not cyclic)
  * edge cases:
    * edge cases are tested by unit tests (HUnit)
  * conform doc.:
    * all tests are conform to documentation, e.g. source code comments by haddock)

After checking the above mentioned criteria, the result is documented as source code comment:
  * <code>
{-  * validated: ✅
        * completeness: ✅
        * independence: ✅
        * edge cases  : ✅
        * conform doc.: ✅ -}
tgUnitf1 :: T.TestTree
tgUnitf1 =
    T.testGroup
        "f1"
        [
            -- ...
        ]
</code>

==== Classes ====

Each function has its own test group designated as ''tgClass<ClassName>''.

Tests are validated by checking the following criteria:
  * documented:
    * laws:
        * laws and its criteria are described and are verifyable
     * characteristics:
        * the general characteristics of the class are described
    * prefix
        * the prefix of the class is given
    * short description
        * a short description is given
  * completeness:
    * the class is tested to fulfill all laws
    * the class is tested for all types and type combinations, respectively
    * and the class is tested for all types and type combinations
  * stubbed:
    * if any, the default functions of the class are also tested by stub instance(s), as well

After checking the abovementioned criteria, the result is documented as source code comment:
  * <code>
{-  * validated: ✅
        * documented: ✅
          * laws             : ✅
          * characteristics  : ✅
          * prefix           : ✅
          * short description: ✅
        * completeness: ✅
            * all laws documented          : ✅
            * all type combinations        : ✅
            * stubbed for default functions: ✅ -}
tgClassA :: T.TestTree
tgClassA =
    T.testGroup
        "class A"
        [
            -- laws:
            tgLawA,
            tgLawB,
            tgLawC,
            -- unit tests:
            tgUnitf3,
            tgUnitlcdConvert
        ]
</code>

==== Modules ====

Each module exports a test group designated as ''testGroup''. 

Tests are validated by checking the following criteria:
  * documented:
    * there is completely and correct filled header like: <code>
Description : provides a class for ...
Copyright   : (c) <Author>, <Year>
License     : <License>
Maintainer  : <E-Mail-Address>
Stability   : experimental
Portability : POSIX
</code>
    * It follows a general description of the purpose of the module.
  * completeness:
    * all classes are in the root test group
    * all global functions have tests in the root test group
    * all classes are validated
    * all global functions are validated

  * example: <code>
{-  * validated: ✅
        * documented: ✅
        * completeness: ✅ -}
testGroup :: T.TestTree
testGroup =
    T.testGroup
        "module A"
        [
            tgClassA,
            tgUnitf1,
            tgUnitf2,
            tgUnitf3
        ]
</code>


===== ✎ =====
~~DISCUSSION~~