What is JSONL?

What is JSONL

JSONL stands for JSON Lines, a text format that stores one JSON object per line. This format is useful for streaming data, processing large files, and working with datasets that contain different types of objects.

In this blog post, I will explain the advantages of JSONL over regular JSON, how to create and read JSONL files, and some common use cases for this format.

JSON vs JSONL

JSON (JavaScript Object Notation) is a popular data interchange format that is easy to read and write for humans and machines. It is based on a subset of the JavaScript language and can represent simple data structures and associative arrays (called objects).

A typical JSON file looks like this:

{

"name": "Alice",

"age": 25,

"hobbies": ["reading", "biking", "cooking"]

}

However, JSON has some limitations when it comes to storing large or heterogeneous data. For example, if you want to store a collection of objects, you have to wrap them in an array, which adds an extra level of nesting and makes the file harder to parse. Also, if you want to append new objects to an existing JSON file, you have to rewrite the whole file, which can be inefficient and error-prone.

JSONL solves these problems by storing one JSON object per line, without any wrapping or commas. Each line is a valid JSON value that can be parsed independently. A JSONL file looks like this:

{"name": "Alice", "age": 25, "hobbies": ["reading", "biking", "cooking"]}

{"name": "Bob", "age": 30, "hobbies": ["gaming", "coding", "traveling"]}

{"name": "Charlie", "age": 35, "hobbies": ["music", "photography", "writing"]}

As you can see, JSONL has several advantages over JSON:

- It is more compact and readable, as it eliminates unnecessary brackets and commas.

- It is more flexible and scalable, as it can store different types of objects without requiring a common schema or structure.

- It is more efficient and robust, as it can handle streaming data, append new objects without rewriting the file, and recover from partial or corrupted data.

How to create and read JSONL files

There are many tools and libraries that can help you create and read JSONL files in various programming languages. Here are some examples:

- In Python, you can use the jsonlines module to read and write JSONL files. For example:

import jsonlines

# write to a JSONL file

with jsonlines.open('data.jsonl', mode='w') as writer:

writer.write({"name": "Alice", "age": 25, "hobbies": ["reading", "biking", "cooking"]})

writer.write({"name": "Bob", "age": 30, "hobbies": ["gaming", "coding", "traveling"]})

writer.write({"name": "Charlie", "age": 35, "hobbies": ["music", "photography", "writing"]})

# read from a JSONL file

with jsonlines.open('data.jsonl') as reader:

for obj in reader:

print(obj)

- In JavaScript, you can use the jsonlines-stream module to read and write JSONL files. For example:

const fs = require('fs');

const jsonlines = require('jsonlines-stream');

// write to a JSONL file

const writer = fs.createWriteStream('data.jsonl');

writer.write(JSON.stringify({"name": "Alice", "age": 25, "hobbies": ["reading", "biking", "cooking"]}) + '\n');

writer.write(JSON.stringify({"name": "Bob", "age": 30, "hobbies": ["gaming", "coding", "traveling"]}) + '\n');

writer.write(JSON.stringify({"name": "Charlie", "age": 35, "hobbies": ["music", "photography", "writing"]}) + '\n');

writer.end();

// read from a JSONL file

const reader = fs.createReadStream('data.jsonl').pipe(jsonlines.parse());

reader.on('data', (obj) => {

console.log(obj);

});

- In R, you can use the jsonlite package to read and write JSONL files. For example:

library(jsonlite)

# write to a JSONL file

data <- list(

list(name = "Alice", age = 25, hobbies = c("reading", "biking", "cooking")),

list(name = "Bob", age = 30, hobbies = c("gaming", "coding", "traveling")),

list(name = "Charlie", age = 35, hobbies = c("music", "photography", "writing"))

)

write_json(data, 'data.jsonl', ndjson = TRUE)

# read from a JSONL file

data <- read_json('data.jsonl', simplifyVector = TRUE)

print(data)

Use cases for JSONL

JSONL is a versatile format that can be used for various purposes, such as:

- Storing and processing large datasets that do not fit in memory or have a fixed schema.

- Streaming data from web APIs, sensors, logs, or other sources that produce JSON objects.

- Exchanging data between different applications or systems that support JSON.

- Performing data analysis or machine learning tasks that require reading or writing JSON objects.

Conclusion

JSONL is a simple and powerful format that extends the capabilities of JSON for storing and processing large or heterogeneous data. It is easy to create and read with various tools and libraries, and it can handle streaming data, appending data, and recovering data. If you are looking for a flexible and efficient way to work with JSON data, you should consider using JSONL.