By Elson Correia
In order to teach myself my ways of working with NodeJs, Javascript, and Typescript, I have built many versions of a file-based JSON database over the years, especially when I did not want to settle on a database setup for my project.
For this tutorial, I’ll show you how to put a file-based JSON database together quickly and enjoy every related algorithm. We will look into:
- Using file system module to read-write files;
- Create a powerful query API for the data;
- Handle data CRUD actions;
- Analyse ways to improve the data read-write performance;
Start a Typescript Node Project
I wrote a post on how to set up a Typescript Node project and shared it as a template you can clone and start a Node project. That’s the environment I will be using for this tutorial so simply run the following command to create your project:
git clone git@github.com:beforesemicolon/node-typescript-project-template.git
Make any necessary changes to the project and, let’s get going!
Implementation
I always start anything by walking backward from the developer experience. Then, I write the code for the behind-the-scenes to support those interactions. I call it DX-driven development.
I picture the database being started like so:
interface ToDo {
id: string,
name: string;
status: "pending" | "completed" | "archived";
user: {
name: string;
};
}
const db = new JSONDB<ToDo>("todo");
It should simply take the name of the JSON file where all the data will be placed and take a generic type which should work like a schema of the data to help with the developer experience. The code behind this should look like so:
// src/db/JSONDB.ts
import path from "path";
import fs from "fs";
class JSONDB<T extends object> {
private readonly filePath: string;
constructor(private fileName: string) {
this.filePath = path.join(__dirname, `${fileName}.json`);
if (!fs.existsSync(this.filePath)) {
fs.writeFileSync(this.filePath, "[]", "utf-8");
}
}
}
The class is generic and the item T extends object and for a JSON file, it narrows to Object literal and Array.
Look that I used the existsSync and writeFileSync methods of the file system. This is because I want the file to be created with the class instantiation synchronously which is a blocking action. An alternative would be to have a type of init method that is asynchronous where such action can be performed. It’s up to you!
Now let’s look at some CRUD actions starting with the insert . On the usage side, I picture someone doing something like so:
const todo = await db.insert({
id: crypto.randomUUID(),
name: "Go To Gym",
status: "pending",
user: {
name: "John Doe",
}
});
The insert action should be straightforward. It only takes the data I want to put in my DB and returns it back. The database should not handle any validation or data formatting. That’s up to the dev but with Typescript, the data type should be checked, which is great!
Every CRUD action needs to be asynchronous and I should be able to
awaitany actions.
Before we add the insert method, I need the ability to read and write to the JSON file. For those actions, I will add the read and write methods and make them private.
// src/db/DB.ts
import path from "path";
import fs from "fs";
import { writeFile, readFile } from "fs/promises";
class JSONDB<T> {
...
private async read(): Promise<Array<T>> {
return JSON.parse(await readFile(this.filePath, "utf-8"));
}
private async save(data: T | Array<T>) {
try {
let content = data;
if (!Array.isArray(data)) {
content = await this.read();
content.push(data);
}
return writeFile(this.filePath, JSON.stringify(content));
} catch (e) {
console.error(e);
throw new Error("Failed to save data to file");
}
}
}
The write method should either take a single item or a list of them to save to the JSON file. This is because sometimes I may need to filter the list to remove items or simply append them to the list. This will make sense as we add more methods.
With save and read methods in place, our insert method should look like so:
// src/db/DB.ts
...
class JSONDB<T> {
private _size = 0;
private readonly filePath: string;
get size() {
return this._size;
}
...
async insert(data: T) {
await this.save(data);
this._size += 1;
return data;
}
...
}
I made the deliberate choice to not handle the error here. If I did, it would be just to throw it back with a nicer message but I will leave the error to be handled where the method was called. Feel free to change that.
Notice that I also introduced a size getter and a private _size property which I will be updating with many other methods which change the data list size.
Now…
To be able to get items, I should be able to chain “matchers” for the data I want. For example:
await db.getOne()
.where("status").equals("completed")
.where("user.name").equals("John Doe")
.run();
The where should be enough and should take the key or dot-separated keys for the DB to match on. I should be able to chain as many matchers as I want, and they should allow me to perform equals , lessThen , greaterThen , lessOrEqual , greaterOrEqual and many other matches.
I should also be able to specify the keys of the data I want back by comma-separating the keys and key chains as arguments of the action called (in this case getOne).
const item = await db.getOne("name", "user.name")
.where("status").equals("completed")
.where("user.name").equals("John Doe")
.run();
/* item will look like:
{
name: "Buy Groceries",
user: {
name: "John Doe"
}
}
Here is where this implementation gets fun! Let’s break down the sub-problems when it comes to supporting this interface/experience:
- I need a way to take dot-separated keys and index and deep read values of an object. For example,
user,user.name, anduser.items.0.nameare all valid key chains; - I need to chain
wherematchers on every action and chain comparator to everywherematcher. For example,where.equalsandwhere.lessThenare valid butwhereby itself is not. - I need a way to determine when the chaining ends and run the query. This will be done with the method
runwhich is at the same level aswhere. - This entire operation is asynchronous except for the matchers. That means that only
runis anasyncmethod.
I am pretty sure there will be sub-problems of these sub-problems that we will need to solve. I am using a programming technique I spoke about in my article “How to Solve any Programming Problem” which you can check later.
Let’s solve them…
First, let's transform key chains into a value:
// src/utils/get-key-chain-value.ts
import { ObjectLiteral } from "../types";
export function getKeyChainValue(
keyChain: string,
data: ObjectLiteral
): unknown {
const parts = String(keyChain).split(".");
const key = parts.shift() as string;
const value = data[key] as ObjectLiteral;
if (parts.length) {
if (value && typeof value === "object") {
return getKeyChainValue(parts.join("."), value);
}
throw new Error(`Cannot get "${parts.join(".")}" of ${value}`);
}
return value;
}
Here I am treating data as a ObjectLiteral but it can be an object literal or an Array, the only objects allowed in a JSON file.
This is a recursive method that takes the first key in the chain, gets the value, and returns it as long as there are no more keys to go through. If there are keys to go through and the current value is an “object” we call getKeyChainValue method again, otherwise throw an error because you can’t drill deeper.
Now let’s handle the where …
I picture it being a separate function that takes two callbacks. One to collect the matchers and another to run at the end.
// src/utils/where.ts
export const where = <T>(
collector: (key: keyof T | string, comparator: Comparator, value: unknown) => void,
runner: () => Promise<void>
) => {
// logic here
}
The Comparator is simply an enum with all types of comparisons we want to make.
enum Comparator {
Equals,
NotEqual,
In,
Between,
GreaterThen,
LessThen,
GreaterOrEqual,
LessOrEqual,
Matches,
}
Let’s look at the usage format to understand the experience when calling where .
action().where(KEY_OR_KEY_CHAIN).COMPARATOR(VALUE)
// real example
db.getOne().where("status").equals("completed")
db.getAll().where("user.name").matches(/Doe$/)
We need to know the key or key chain of the data ( where argument), which type of comparison we will need to do ( equals , lessThen , etc) which we are calling comparator and then the value we are comparing against.
// format
KEY_VALUE(extracted from KEY) COMPARATOR VALUE
// real example
"pending" == "completed"
30 < 45
Before we implement the body of the where let’s handle the matcher collecting which will help us understand it further:
// src/utils/collect.ts
import { Matcher } from "../types";
import { where } from "./where";
export function collect<T, R>(done: (res: Matcher<T>[]) => R) {
const matchers: Matcher<T>[] = [];
const run = async () => done(matchers);
return {
where: where<T, R>((key, comparator, value) => {
matchers.push({ key, comparator, value });
}, run),
run,
};
}
What the above code is saying is that every time we complete a matching chain like where(KEY).COMPARATOR(VALUE) the collector is called and in the above case, it simply pushes a matcher to the list. Once the run is called we call the done function which will handle the logic of using the matchers against the data to perform whatever action.
By the way, the Matcher is an interface for the data collected by the where function that looks like so:
interface Matcher<T> {
key: keyof T | string;
comparator: Comparator;
value: unknown | string | RegExp | [number, number] | unknown[] | number;
}
Now let’s implement the where function …
// src/utils/where.ts
import { Collector, Comparator } from "../types";
export const where = <T, R>(
collector: Collector<T>,
runner: () => Promise<R>
) => {
return (key: keyof T | string) => {
const chain = {
where: where<T, R>(collector, runner),
run: runner
};
return {
matches(val: string | RegExp) {
collector(key, Comparator.Matches, val);
return chain;
},
equals(val: unknown) {
collector(key, Comparator.Equals, val);
return chain;
},
notEqual(val: unknown) {
collector(key, Comparator.NotEqual, val);
return chain;
},
in(val: Array<unknown>) {
collector(key, Comparator.In, val);
return chain;
},
between(val: [number, number]) {
collector(key, Comparator.Between, val);
return chain;
},
lessThen(val: number) {
collector(key, Comparator.LessThen, val);
return chain;
},
lessOrEqual(val: number) {
collector(key, Comparator.LessOrEqual, val);
return chain;
},
greaterThen(val: number) {
collector(key, Comparator.GreaterThen, val);
return chain;
},
greaterOrEqual(val: number) {
collector(key, Comparator.GreaterOrEqual, val);
return chain;
},
};
};
};
When we call the where function it returns a function — the where function. Thats why the collect method returns an object {where, run} and puts the return function as the where key in that object.
The returned where function takes a key or key chain and then returns an object with all the comparators. Each will call the collector function with the key, the comparator type, and the value they received when called.
Then it does the same thing the collect method does which is to return a new {where, run} object to restart the cycle. The developer can then choose to call the where again or call the run to complete the action.
That’s it!
Get a quickly introduction to recursions by watching this short video
We now have a way to collect the matchers but we still need to handle the comparison. Let’s add a method for it:
// src/utils/match-data-key-value.ts
import { Comparator, Matcher, ObjectLiteral } from "../types";
import { getKeyChainValue } from "./get-key-chain-value";
export function matchDataKeyValue<T>(
data: T,
{ comparator, key, value }: Matcher<T>
) {
const val = getKeyChainValue(key as string, data as ObjectLiteral);
switch (comparator) {
case Comparator.Equals:
return val === value;
case Comparator.NotEqual:
return val !== value;
case Comparator.In:
return (value as Array<unknown>).includes(val);
case Comparator.Between:
return (
Number(val) > Number((value as Array<number>)[0]) &&
Number(val) < Number((value as Array<number>)[1])
);
case Comparator.GreaterOrEqual:
return Number(val) >= Number(value);
case Comparator.GreaterThen:
return Number(val) > Number(value);
case Comparator.LessOrEqual:
return Number(val) <= Number(value);
case Comparator.LessThen:
return Number(val) < Number(value);
case Comparator.Matches:
return typeof value === "string"
? new RegExp(value).test(`${val}`)
: (value as RegExp).test(`${val}`);
}
return false;
}
The matchDataKeyValue method takes the item in our list and the Matcher to compare it against. The first thing we need is to get the value from the key. For that, we use the getKeyChainValue method created earlier.
Then it is down to the type of comparator. That simple!
Try to come up with your own comparators and matching logic for the database. This setup should allow you to do it easily.
Now let’s handle querying for a single item in the database…
// src/db/DB.ts
...
class JSONDB<T> {
...
getOne(...keys: (keyof T | string)[]) {
return collect<T, Promise<T | Partial<T> | null>>(async (matchers) => {
const item = (await this.read()).find((item) => {
return matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
);
});
if (item) {
return item;
}
return null;
});
}
...
}
The getOne here is calling the collect private method and passing the callback which should be called with all the collected matchers.
Once that is done, we are reading the JSON file and call the find array method for each item we are trying to match all the matchers against by using the array every method to make sure we only use the item which satisfies all matchers.
You can become a PRO with Javascript Array manipulation by watching all my videos on Array data structure.
There is one thing we are not doing here which is to return an object with only the keys requested. For that, let’s first introduce a private method that creates this partial item from the matched items using provided keys.
// src/utils/create-item-from-keys.ts
import { ObjectLiteral } from "../types";
export function createItemFromKeys(
keys: string[],
data: ObjectLiteral
): ObjectLiteral {
const partialItem: ObjectLiteral = {};
keys.forEach((keyChain) => {
let target = partialItem;
let source = data;
String(keyChain)
.split(".")
.forEach((key, idx, parts) => {
const value = source[key] as ObjectLiteral;
// since this is in the context of JSON,
// undefined means the key does not exist
if (value === undefined) {
throw new Error(
`Key "${key}" does not exist in ${JSON.stringify(source)}`
);
}
const isLastKey = idx == parts.length - 1;
target[key] =
target[key] ?? (!isLastKey ? (Array.isArray(value) ? [] : {}) : value);
target = target[key] as ObjectLiteral;
source = value;
});
});
return partialItem;
}
Here we are simply looping the key chains and creating the partial item object key by key. It is a simple tree navigation algorithm that keeps on checking the children of objects until there are no more keys to check or children to drill further.
Now we can update our getOne method to honor the keys requested
// src/db/DB.ts
...
class JSONDB<T> {
...
getOne(...keys: (keyof T | string)[]) {
return collect<T, Promise<T | Partial<T> | null>>(async (matchers) => {
const item = (await this.read()).find((item) => {
return matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
);
});
if (item) {
if (keys.length) {
return createItemFromKeys(
keys as string[],
item as ObjectLiteral
) as Partial<T>;
}
return item;
}
return null;
});
}
...
}
With this final setup, we are ready to handle all the other CRUD methods of this database.
Let’s quickly do that…
Here is the getAll method which uses the array filter method to collect all the matching items instead of the find method.
getAll(...keys: (keyof T | string)[]) {
return collect<T, Promise<T[] | Partial<T[]> | null>>(async (matchers) => {
const items = (await this.read()).filter((item) => {
return matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
);
});
if (keys.length) {
return items.map((item: T) => {
return createItemFromKeys(keys as string[], item as ObjectLiteral);
}) as Partial<T[]>;
}
return items;
});
}
The updateOne method which finds the item and updates it in place to then save the list back into the JSON file.
updateOne(data: Partial<T>) {
return collect<T, Promise<T | null>>(async (matchers) => {
const list = await this.read();
const item = list.find((item) => {
return matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
);
});
if (item) {
mergeObjects(item, data);
await this.save(list);
return item;
}
return null;
});
}
The mergeObjects is a simple utility function that deeply merges object literals and arrays. I shared this in a different article called “25 JavaScript Tricks You Need To Know About (Part 2)”.
The updateAll method which filters the matching items and also updates them in place to then save the list.
updateAll(data: Partial<T>) {
return collect<T, Promise<T[] | null>>(async (matchers) => {
const list = await this.read();
const items = list.filter((item) => {
return matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
);
});
items.forEach((item) => {
mergeObjects(item, data);
});
await this.save(list);
return items;
});
}
The deleteOne method finds the index of the matching item and if the such item exists, it splices it out of the list and saves the list to return the removed item then.
deleteOne() {
return collect<T, Promise<T | null>>(async (matchers) => {
const list = await this.read();
const existingItemIndex = list.findIndex((item) => {
return matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
);
});
if (existingItemIndex >= 0) {
const [existingItem] = list.splice(existingItemIndex, 1);
await this.save(list);
this._size -= 1;
return existingItem;
}
return null;
});
}
The deleteAll method which filters the items out of the list while collecting them and then saves the list back into the JSON file to return the removed items.
deleteAll() {
return collect<T, Promise<T[] | null>>(async (matchers) => {
const existingItems: Array<T> = [];
const list = (await this.read()).filter((item) => {
if (
matchers.every((matcher: Matcher<T>) =>
matchDataKeyValue(item, matcher)
)
) {
existingItems.push(item);
return false;
}
return true;
});
if (existingItems.length) {
await this.save(list);
this._size = list.length;
return existingItems;
}
return null;
});
}
And finally, the ability to drop a database just for good luck.
// src/db/DB.ts
...
class JSONDB<T> {
...
async drop() {
await unlink(this.filePath);
}
...
}
What Next?
This is such a nice project and an excellent one if you are looking to learn more about Javascript and Node together. However, this can be improved way beyond this point. Here are a couple of ideas:
- Introduce more comparators;
- Introduce more matcher types like
.limitto get only limited items;groupByto actually group results; etc. Have fun! - Optimize the database with caching and indexing to make it return results faster and minimize reads and writes to the file.
I provided you with the blueprint to further improve and meet your needs. Now it is up to you!
You can actually install this package and try it out on your project. It is called @beforesemicolon/node-json-db
@beforesemicolon/node-json-db *A simple basic JSON file-based database meant for development mode ONLY. Latest version: 1.0.1, last published: 12…*www.npmjs.com
YouTube Channel: Before Semicolon Website: beforesemicolon.com