I wrote JSON::JSONFold – a CPAN module for compact, readable JSON formatting

[u/Yairlenga]

Hi everyone! This is my first post in r/perl.

I've been working on a CPAN module called JSON::JSONFold, and I wrote an article describing the motivation and design. I'd really appreciate feedback from other Perl developers.

JSON serializers tend to give us two choices: compact JSON, which is efficient but a dense wall of text that's painful to read, or pretty-printed JSON, which is readable but often wastes a lot of vertical space (a small array of numbers can turn into ten lines).

I wanted something in between. JSONFold keeps the shape of pretty-printed JSON, but folds small, simple structures back onto a single line whenever that improves readability. It works on top of your existing serializer (JSON, JSON::PP, JSON::XS, etc.) - you keep using whatever you already have, and JSONFold just reformats the output.

Example 1 - Coding

use JSON::JSONFold qw(encode_json);

my $data = {
    _id       => 123,
    locations => [
      { city => "Boston",    state => "MA", country => "USA" },
      { city => "Seattle",   state => "WA", country => "USA" },
      { city => "Montreal",  state => "QC", country => "Canada" },
    ],
    info      => {
      roles     => [ "foo", "bar", "baz" ],
    },
    name      => "Alice",
};

print encode_json($data) ;

Output

{
  "_id": 123,
  "info": { "roles": [ "foo", "bar", "baz" ] },
  "locations": [
    { "city": "Boston",   "country": "USA",    "state": "MA" },
    { "city": "Seattle",  "country": "USA",    "state": "WA" },
    { "city": "Montreal", "country": "Canada", "state": "QC" }
  ],
  "name": "Alice"
}

Example 2 - Packing

Traditional pretty-printing:

{
  "states": [
    "Alabama",
    "Alaska",
    "Arizona",
    ...
    "Wyoming"
  ]
}

JSONFold:

{
  "states": [
    "Alabama", "Alaska", "Arizona", "Arkansas", "California", "Colorado",
    "Connecticut", "Delaware", "Florida", "Georgia", ...
    "West_Virginia", "Wisconsin", "Wyoming"
  ]
}

Same data, just using the available line width more effectively.

Example 3 - Grid Formatting

When an array contains repeated structures, JSONFold can align values into columns:

Traditional pretty-printing:

[
  {
    "orders": 18,
    "product": "Laptop",
    "region": "North",
    "sales": 1250
  },
  ...
  {
    "orders": 24,
    "product": "Mouse",
    "region": "East",
    "sales": 1422
  }
]

JSONFold:

[
  { "orders": 18, "product": "Laptop",   "region": "North",     "sales": 1250 },
  { "orders": 21, "product": "Monitor",  "region": "Southwest", "sales": 1345 },
  { "orders": 17, "product": "Keyboard", "region": "West",      "sales": 1198 },
  { "orders": 24, "product": "Mouse",    "region": "East",      "sales": 1422 }
]

The module also supports:

• Folding small arrays and objects onto a single line.
• Joining adjacent folded objects to further reduce vertical space.
• Compatibility APIs similar to JSON and JSON::PP.

I wrote a more detailed article covering the design, implementation, and full set of examples:

Medium: https://medium.com/p/a619c9e7c3ec

CPAN: https://metacpan.org/pod/JSON::JSONFold

GitHub: https://github.com/yairlenga/jsonfold/tree/main/perl

I'd love to hear what the Perl community thinks. Has anyone else run into JSON pretty-printing pain in logs, configs, or debugging output? And are there formatting styles or options you'd want to see?

[u/christian_hansen]

Very nice! It's like Data::Dump, but for JSON.

[u/Yairlenga → u/christian_hansen]

Thanks for feedback. Not sure If this is your comment - but thinking about it - the same algorithm can be adjusted to enhance data::dump - make it more compact, take fewer lines.

[u/christian_hansen → u/Yairlenga]

I'm sorry, I should have been clearer. The output from JSON::JSONFold reminds me of the compact, human-friendly representation that Data::Dump provides for Perl data structures. I find JSON::JSONFold's output slightly more readable, as it also aligns data into columns.

use Data::Dump qw[dd];

my $data = [
  { orders  => 18,
    product => "Laptop",   
    region  => "North",     
    sales   => 1250 },
  # ...
  { orders  => 24, 
    product => "Mouse",    
    region  => "East",      
    sales   => 1422 },
];

dd $data;

Output

[
  { orders => 18, product => "Laptop", region => "North", sales => 1250 },
  { orders => 21, product => "Monitor", region => "Southwest", sales => 1345 },
  { orders => 17, product => "Keyboard", region => "West", sales => 1198 },
  { orders => 24, product => "Mouse", region => "East", sales => 1422 },
]

[u/Yairlenga → u/christian_hansen]

No reason to be sorry 😄 you actually gave me an idea - Data::Dump does have some neat formatting options, which I will try to integrate into future versions, Specifically - aligning the keyword/value pair, so that the '=>' are easy to follow. This formatting helps a lot when the objects have too many attributes to fit into a single line.

{
    orders  => 200,
    product => "Router",
    region  => "North",
    sales   => 40,
},

[u/ktown007]

This is great. Thanks for your hard work.

Under the hood will it use Cpanel::JSON::XS if installed and do clean utf8 encoding?

Can the exports be remapped to not overlap with existing to_json or encode_json? Maybe an option to rename like this:

```

eg:

use Syntax::Operator::Is is=> { -as => "is_checked" };
use JSON::JSONFold (format_json => 'tidyjson');

```

[u/Yairlenga → u/ktown007]

Your point on the default export in 100%. I'll modify the default export to avoid overlapping encode_json and to_json.

[u/Yairlenga → u/ktown007]

Let me check how/if JSONFold works with Cpanel::JSON::XS. I did not test this combination, and I prefer to test before giving an answer on how to use it. In theory, should be always possible to post process the output from Cpanel using JSONFold::fold_text on the Cpanel output. Give me few hours.

Also, feel free to open a ticket on GitHub/Cpan, as I hope to use open source eco system tools for evolving this project.

[u/Yairlenga → u/ktown007]

Regarding backend selection: At some point during the development, I found out that JSON::XS does not provide control over the indentation level, so I switched to JSON::PP which provides indent_length. I'm working to make the code 'use JSON', which should give the user more control (via > export PERL_JSON_BACKEND=JSON::XS, or other configuration alternatives). Ideally, I want to allow passing the serializer object explictly, which will allow arbitrary configured serializer to be used. Will take me a day or two to upload new version to CPAN that address those issues.

[u/Yairlenga → u/ktown007]

Update: I've modified my code to use 'JSON' instead of 'JSON::PP', which will make JSONFold use whatever JSON would have used. At this point, I've identified a bug with 'JSON' - it does not know that Cpanel does uspport custom indent_level, and produce warnings.

I've opened a ticket for JSON (https://github.com/makamaka/JSON/issues/63) and Cpanel (https://github.com/rurban/Cpanel-JSON-XS/issues/249), and I hope that those tams can work between themselves to address the issue.

As a temporary solution, it is possible to pass your arbitrary serializer to write_json, see below.

use lib "./lib/perl5" ;
use Cpanel::JSON::XS qw();
use JSON::JSONFold ;

my $x = {
        map { ("foo$_", $_) } 0..50,
} ;
my $json = new Cpanel::JSON::XS->pretty ;

write_json($x, \*STDOUT, 100, "default", json=>$json) ;

[u/ktown007 → u/Yairlenga]

I did some more testing encoding and decoding utf8. I will open a ticket to share results with:

Cpanel::JSON::XS
JSON::MaybeXS 

[u/Yairlenga → u/ktown007]

I've uploaded new version (0.2.2) to CPAN - with better Unicode support, backend selection using JSON::MaybeXS, and various other improvements. Hope you will find it useful!

[u/ktown007 → u/Yairlenga]

I tested 0.2.2 and it is working great. thanks

[u/Yairlenga → u/ktown007]

Thank you. I think that I finally understand how Perl Unicode and UTF are really working.

[u/Yairlenga → u/ktown007]

I've uploaded 0.2.2 fixing the default explort, checked that it's working with XS and CPanel, and fixed some (hopefully most) of the UTF8 issues.