Testing JSON grammars

Perl 6 grammars make it easy to parse JSON and it’s a favorite example to show off the feature. This post is about testing the grammar; you don’t need another example here. I list several at the end of this post.

Standardized Test Close-Up

I went looking for some sort of official test corpus to throw at parsers. Googling gave me some internal links to json.org for a tool called JSON_Checker that has some test files. As I created my own JSON grammar (which looks just like all the examples I list at the end of this post, really), I used JSON_Checker files as the tests. I fixed some tests and put my versions in it my json-acceptance-tests GitHub repo.

use Terminal::ANSIColor;

# https://github.com/briandfoy/json-acceptance-tests
my $pass_fail_directory = @*ARGS[0]; # wherever you put them
my @files = dir( $pass_fail_directory, test => rx/ '.json' $ / );

# two grammars I created, which aren't important here.
my @grammars = <Grammar::JSON Grammar::RFC7159>;

for @grammars -> $grammar {
	put "Trying grammar $grammar";
	require ::($grammar);
	try-grammar( $grammar, @files );
	put "\n";

sub try-grammar ( $grammar-class, @files ) {
	state @tag = (
		colored( 'NOT OK', 'white on_red'   ),
		colored( '  OK  ', 'white on_green' ),
	state %expected = (
		'pass' => 0,
		'fail' => 1,

	for @files -> $file {
		$file.IO.basename ~~ m/ ^ $<result> = [ pass | 'fail' ] $<number> = \d+/;
		my ( $expected, $number ) = $<result>, $<number>;
		my $result = ::($grammar-class).parsefile( $file );

		say @tag[ (%expected{$expected} + $result.so) % 2 ] ~ " { $file.IO.basename }";

Here’s the output, which I’ve labeled nicely with Terminal::ANSIColor. Those failures aren’t real. One doesn’t believe that anything other than an object or array can be a top-level value and the other thinks you should stop parsing at a certain depth.

There’s a fake REST service and some canned files for popular services I could test against too. I do wish the RFCs came with test suites and sample data though. If you know of other useful collections of valid and invalid files, tell me about them.

If you want some grammar examples, here are some JSON parsers:

Leave a Reply

Your email address will not be published. Required fields are marked *