Does it make any sense to use trivial (i.e. constant/nonvarying) generators for types that functions/classes under test are parametric with respect to?
the type system already guarantees the function can't screw up by examining the values of those types, so you can basically put unit in there and still discover any bugs there are to find, no?
e.g. if testing the functor laws for (a -> b) -> f a -> f b, they'll use integers for a and b. what I'm wondering is if its' safe for me to be stupider and just use (). will i still uncover whatever bugs there are to be found?
Isn't it that for such type of testing, we want "counting" generator? We don't really care about the content, we care about it's uniqueness and position in resulting value (you can only move value around and apply arguments to it if it's polymorphic, can't you?). Simply spitting out unique values would uncover "moving around" immediately, while random generator could potentially need more runs in case it "accidentally" generates same values in multiple places.
very interesting. ok, so now that you've invalidated that hypothesis, how about the "dual" hypothesis: does using a generator that never produces the same thing twice for those type parameters work?
everything here is polymorphic! but if i have (badly) implemented insert in such a way that it only works once. in this case, a constant generator will say "yup, everything is fine!"
let's assume you're right that parametricity solves this problem for us! even then, you're also going to want to test code that is concrete, where a deterministic generator can always be detected, and then hide functionality or w/e
Seems like this is the weakness of not having dependent or refinement types to serve as proofs. Property tests generate values so that we can assert invariants on the domain and codomain of our functions which types alone are not able to express, no?
I think polymorphism alone is not sufficient to describe such properties otherwise we would be able to prove a functor is valid in Haskell alone, wouldn't we?
Does it make any sense to use trivial (i.e. constant/nonvarying) generators for types that functions/classes under test are parametric with respect to?
the type system already guarantees the function can't screw up by examining the values of those types, so you can basically put unit in there and still discover any bugs there are to find, no?
is there something prompting you to suspect otherwise?
Just my inexperience with property based testing :)
most of the "xyz-classes" collections of laws I've seen online use something like an integer for types that the class is parametric wrt
e.g. if testing the functor laws for
(a -> b) -> f a -> f b
, they'll use integers fora
andb
. what I'm wondering is if its' safe for me to be stupider and just use()
. will i still uncover whatever bugs there are to be found?well.. I would still use something with more than one value
Ok, why is that?
if you want to check
fmap f . fmap g $ xs == fmap (f . g) xs
you would need to use somef
, and if thatf
is alsoIdentity
it would matterbecause
Identity ()
has only one (not containing bottom) value, so it will always be==
to itselfthe
fmap
implementation never analyzes thea
orb
, that's guaranteed by the type system. so why would it matter whether I have aBool
or a()
?right. and there is actually only one way to write the fmap function for
Identity
just based on the types I thinkIf you erase the newtype, that's:
I don't think it's possible to screw that up and still typecheck
so you wouldn't be testing it for
Identity
anywaybut I think this argument can be transferred to something more complex, e.g.
Maybe
?hmm. with
Maybe
can't I just returnNothing
for theJust
case and screw up?if I have that bad implementation, and I'm only ever using
()
s for the type parameter toMaybe
, i think I'd still catch itLike I said, I actually don't know whether this generalizes, that's why i'm asking. It just sounds like something that might make sense.
I'd also wager on "it doesn't matter", I'm just trying to "play devil's advocate"
yes, that's exactly what i'm looking for, thanks
e.g. does something go wrong if we apply this principle to
Traversable
? IsIdentity
the only applicative I need to test with?for this using
[()] -> Maybe ()
will never be able to catch that this implementation is "wrong":whereas
[Integer] -> Maybe Integer
has a chance of doing sowhat is wrong with it?
like, what's the law we're testing?
wrong in the sense that it doesn't do the same thing as
Data.Maybe.listToMaybe
:Nothing
so the law requires an
Eq
constraint ona
?yes
like the
fmap
onehmm ok. I guess that's a good counterexample.
i'm having some trouble figuring out how to write the law
listToMaybe [] = Nothing
is law 0that's the problem with the example - the law is the implementation itself, basically
listToMaybe (x : xs) = Just x
i guess that's the second one?
this gives us a counterexample for the fmap thing as well I believe
for
fmap :: (a -> b) -> [a] -> [b]
I meanlike so?
hmm, i'm not sure we can actually write that second case
oops
ah, ok
yeah, exactly. so that one would be fine if we were looking at
()
s all the timeIsn't it that for such type of testing, we want "counting" generator? We don't really care about the content, we care about it's uniqueness and position in resulting value (you can only move value around and apply arguments to it if it's polymorphic, can't you?). Simply spitting out unique values would uncover "moving around" immediately, while random generator could potentially need more runs in case it "accidentally" generates same values in multiple places.
very interesting. ok, so now that you've invalidated that hypothesis, how about the "dual" hypothesis: does using a generator that never produces the same thing twice for those type parameters work?
@TheMatten you beat me to it!
(using
Const ()
for testingTraversable
s should also exhibit the same flaw, I think)then you can't test the property of "being a function"
because you'll never encounter the same
x
twicegranted, that is a very weird thing to test
unsafePerformIO
stuff may need more than property testing :sweat_smile:not only for actual functions:
if you've implemented
Map k v
as[(k, v)]
under the hood, you may want a property that for allMap
s, for eachk
there is at most onev
this thread is too long for me to work through, so at the risk of repeating something already said
no, you still want randomness!
maybe i am implementing a set and property testing it
probably i want a law that says
insert a (insert b s) == insert b (insert a s)
everything here is polymorphic! but if i have (badly) implemented
insert
in such a way that it only works once. in this case, a constant generator will say "yup, everything is fine!"there is a trivial model where you map everything to
()
; using a constant generator can't differentiate your model from that one@Sandy Maguire What about the non random generator that always generates distinct values discussed above (e.g. natural numbers 0..infinity)?
would that fail to detect an unlawful implementation in your set example?
then you don't get automatic testing of crazy values
maybe my code only works up to 100 or something?
Your code doesn't have the opportunity to only work up to 100
because it is parametric
not true
well, i guess true if you don't have any constraints
but constraints can do arbitrarily crazy things :)
If it isn't parametric in the relevant type then it's not relevant to what we're talking about
if you're talking fully
forall a. a -> ...
then yeah, sure, it's fineexcept if your code doesn't work now if you get two of the same value?
I don't quite understand that last bit
trying hard to construct an example
maybe it can't be done without an
Eq
I'm talking about this bit. I don't quite understand what this means.
okay well let's assume i have an eq on
a
i have a bad impl:
maybe this can't exist without an eq constraint, so maybe you're fine
let's assume you're right that parametricity solves this problem for us! even then, you're also going to want to test code that is concrete, where a deterministic generator can always be detected, and then hide functionality or w/e
Seems like this is the weakness of not having dependent or refinement types to serve as proofs. Property tests generate values so that we can assert invariants on the domain and codomain of our functions which types alone are not able to express, no?
They're not proofs but they give us some amount of certainty that for a sampling of the domain our invariant holds.
I think polymorphism alone is not sufficient to describe such properties otherwise we would be able to prove a functor is valid in Haskell alone, wouldn't we?