> I won't meta-program for tests, but I will do things like make a list of classes, or symbols or whatever, to pass to a loop. Just keep it simple.
This is a case where metaprogramming for tests came in handy.
I had a bug recently that involved someone making a change that violated an invariant property of a class. To codify this invariant, I was tempted to do what you did, to make a list of symbols to feed into my test to ensure the invariant was obeyed. However this bug was caused precisely by someone adding a new symbol, a method, that didn't obey this property. The test using this design wouldn't catch this failure. I instead opted to do some introspection (it's Python, so it was dead simple) on the class to ensure all of its methods obeyed this invariant. It took a little extra time to implement but in the end it worked.
With the caveat that I haven't invested the full amount of time necessary to strongly state this opinion publicly, I do feel that metaprogrammed tests are a case of adding abstraction unnecessarily (which, incidentally, is why I haven't taken the time to try to extensively use them on a project.)
A test should be dead simple to read and understand when someone else new to the project needs to understand what it tests. Further, when a test fails, the output should clearly indicate exactly on what line of code an error occurred.
Metaprogramming tests feels to me like a case of a desire for or predilection for cleverness getting in the way of what the task is actually for.
Tests should tell a story. They should not be subjected to the same methods of abstraction used in the code they themselves are supposed to test and verify.
This is a case where metaprogramming for tests came in handy.
I had a bug recently that involved someone making a change that violated an invariant property of a class. To codify this invariant, I was tempted to do what you did, to make a list of symbols to feed into my test to ensure the invariant was obeyed. However this bug was caused precisely by someone adding a new symbol, a method, that didn't obey this property. The test using this design wouldn't catch this failure. I instead opted to do some introspection (it's Python, so it was dead simple) on the class to ensure all of its methods obeyed this invariant. It took a little extra time to implement but in the end it worked.