@Are0h Yeah! This is basically what we used to have with the ActivityPub -- an app that you connected *your* software to as a dev and it told you what about its implementation was broken.

Getting a new test suite up and running is a goal of the W3C SWICG committee but in my experience these things need concerted funds thrown at them to happen