Table of contents
While working on an Express application recently, I came across a very weird bug which honestly humbled me to a great extent. Every time I think I understand the monstrosity that JavaScript is, I'm proved wrong. Let's dive into this!
What happened?
Well, I am currently working on an Express server which is written in TypeScript. I am following a domain-driven folder structure, wherein, I have divided all the code into folders corresponding to the business domain to which the APIs belong. For example, say I have APIs that belong to Screen A of my application, then all the controllers, routers, and services for all APIs used by Screen A, will be inside the screenA_domain folder.
Here's a better visualization of this folder structure:
└── 📁my_server
└── app.ts
└── 📁a_domain
└── 📁services
└── a.ts
└── 📁v1
└── 📁controllers
└── a.ts
└── 📁routes
└── a.ts
└── 📁b_domain
└── 📁services
└── b.ts
└── 📁v1
└── 📁controllers
└── b.ts
└── 📁routes
└── b.ts
└── 📁c_domain
└── 📁services
└── c.ts
└── 📁v1
└── 📁controllers
└── c.ts
└── 📁models
└── c.ts
└── 📁routes
└── c.ts
This folder structure was chosen for a variety of reasons, which we will not be exploring today. However, the root of the bug was this folder structure.
As a consequence of having an individual router for each domain, it would become quite a task to remember to add each domain's router to the main app.ts file. So, as any self-respecting developer would, I spent an hour solving a problem that did not exist :-)
I wrote a function called includeAllDomainRoutes that scanned the parent directory and dynamically added all the routes it found inside any folder ending with the _domain suffix. The function looks like this:
function includeAllDomainRoutes(app: Express) {
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const domainFolders: string[] = fs
.readdirSync(__dirname)
.filter((folder) => /^[a-z]+_domain$/.test(folder));
domainFolders.forEach((domain: string) => {
const versionFolders: string[] = fs
.readdirSync(path.join(__dirname, domain))
.filter((folder) => /^v\d+$/.test(folder));
versionFolders.forEach((version: string) => {
const domainPath: string = path.join(
__dirname,
domain,
version,
'routes'
);
const routeFiles: string[] = fs.readdirSync(domainPath);
routeFiles.forEach(async (file: string) => {
const filePath: string = path.join(domainPath, file);
const importedModule = await import(filePath);
const route: Router = importedModule.default;
app.use('', route);
});
});
});
}
I made sure the function recognized all version folders inside the domain folder and all the route files inside each version folder. However, a very subtle bug crept in.
Some routes were not being added to the app! After I triple-checked my regex-matching and had wasted an hour or so putting in a console.log() after every statement, I had lost all hope! But then, I did the unimaginable... I used the debugger.
Why was it happening?!
Every time I use the debugger I realize how easy it makes debugging... and then I forget about it for the next 3 months!
Coming back to the issue, while using the debugger I noticed that all the domain folders were indeed being recognized by the function. However, for some reason one of the domains' routes was not being registered to the app. The issue seemed to be happening at the following line:
const importedModule = await import(filePath);
On every iteration of the routeFiles.forEach() loop, it was this line that threw the code awry. Every time the debugger reached this line and I instructed it to go to the next one, it for some reason went straight to the next iteration of the loop!
The good news was, that it was a very simple line of code. It was doing nothing but importing the ES module at filePath. The bad news was, that it was such a simple line of code! What could possibly be going wrong?
Sometimes I'm surprised how much the debugging process mirrors life. As in life, when one is stuck in a rut or is just having the worst day of their life, it helps to zoom out and look at the bigger picture. And that's what solved this issue for me too.
Zoom out:
routeFiles.forEach(async (file: string) => {
...
const importedModule = await import(filePath);
...
});
Three fundamental functions are being used here: forEach, await, and import. The first two are concepts that JavaScript developers learn in the first few weeks of their coding journey. The third is not used that often in the middle of a function, instead, people are used to seeing the import statement at the top of a file.
And yet, it was the combination of the first two functions, which caused all the trouble! After spending a few minutes going down the StackOverflow rabbit hole, I found the issue! Turns out, our problem lies in the very implementation of the forEach function.
For those of who don't know, the forEach function is syntactical sugar for a basic for loop. In fact, implementing it yourself is extremely simple!
Array.prototype.myForEach = function myForEach(callback) {
for (let i = 0; i < this.length; i += 1) {
if (Object.hasOwnProperty.call(this, i)) {
callback(this[i], i, this);
}
}
};
That's it!
Now if we pay close attention to how the callback is called inside the loop, we can see that the await keyword is nowhere to be found, which makes sense because you wouldn't have to use it for synchronous code. So, what would happen if your callback is an async function? All of the async callback functions do generate a promise, but we throw them away instead of awaiting them!
And now we can understand why the debugger was acting the way it was. We were being sent to the next iteration of the loop because the promise returned by the async code inside the forEach loop was thrown away.
The Solution
Fixing this problem was as simple as replacing the forEach loop with a simple for loop, and declaring the includeAllDomainRoutes function to be async. Here's the updated code:
async function includeAllDomainRoutes(app: Express) {
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const domainFolders: string[] = fs
.readdirSync(__dirname)
.filter((folder) => /^[a-z]+_domain$/.test(folder));
for (const domain of domainFolders) {
const versionFolders: string[] = fs
.readdirSync(path.join(__dirname, domain))
.filter((folder) => /^v\d+$/.test(folder));
for (const version of versionFolders) {
const domainPath: string = path.join(
__dirname,
domain,
version,
'routes'
);
const routeFiles: string[] = fs.readdirSync(domainPath);
for (const file of routeFiles) {
const filePath: string = path.join(domainPath, file);
const importedModule = await import(filePath);
const route: Router = importedModule.default;
app.use('', route);
}
}
}
}
The abstractions created to enhance the developer experience almost always make us appreciate how well JavaScript has evolved as a language. At the same time, situations like these do make me reconsider my career choices!
If you wish to avoid such revelations yourself, for the sake of your mental health please remember: forEach (a)waits for None!