Navigating the JEST Memory Issue - My Workaround

Let's dive right into the issue our team was facing. We rely on one of the most popular JavaScript testing frameworks, JEST, for rigorously testing our backend APIs. Initially, everything was sailing smoothly - setup was a breeze, and our tests were smooth to run. We were okay with how things were going.

However, as our codebase grew and we added more APIs to the code, problems started cropping up. The test suite began taking longer to complete, and memory consumption started to skyrocket. Our initial solution was to simply bump up the heap memory using --max-old-space-size. It worked like a charm, at least for a while.

As time passed, we kept increasing the heap memory allocation until we reached a point where we realized this band-aid approach wasn't sustainable. We were no longer able to run tests on our local machines, and even in our pipeline, tests were failing due to mysterious memory issues.

That's when we decided it was time to tackle this head-on.

Before diving into this, I checked the Jest GitHub repository for similar issues. To my surprise, there were several reported, but none of the proposed solutions seemed to do the trick for us.

So, I embarked on a trial-and-error journey.

Uncertain if the issue lay with our code or with Jest itself, I decided to start by refactoring some code and tests. After rerunning the tests, I waited for about 12 minutes, only to see no improvement in memory usage. By the way, you can monitor memory usage in Jest using the --logHeapUsage argument.

Feeling a bit lost, I searched for more blogs and videos and came across the term "Memory Leak." With this newfound knowledge, I began searching for these leaks, feeling like a detective using Chrome DevTools to analyze heap memory usage heat maps. I zoomed in and out of those charts repeatedly, but ultimately, I was tired and still clueless.

Back to the GitHub issues, I went, reading comments one after another, still finding no definitive solution. It dawned on me that maybe it wasn't our problem; perhaps it was just how Jest operated. If it were a Jest issue, my initial reaction was, "Let's switch to another testing framework!" But the reality of rewriting the entire testing suite hit us like a ton of bricks, especially with a small team. We were only weeks away from rewriting our front end from React with Material-UI to Vue.js with Tailwind, and we had to fix bugs and implement new features. So, I decided to give it a few more tries before throwing in the towel.

This time, I ran individual tests one by one and then attempted to run everything together. What I noticed was that when I ran tests individually, they ran smoothly, and memory usage stayed within limits. However, when more files were included, memory consumption began to swell. Jest seemed to increase heap usage after each test. Although I didn't know the root cause, I had a workaround. Since I discovered this issue while running tests individually, I thought, "Why not do this every time I need to test something?"

So, I wrote a shell script that looked something like this:

base_dir=""
integration_test_dir=${base_dir}tests/integrationTests
integration_directory_list=($(ls ${integration_test_dir}))

for dir in "${integration_directory_list[@]}"; do
    echo "$dir"
    npm run test ${integration_test_dir}/${dir}
done

unit_test_dir=${base_dir}tests/unitTests
unit_directory_list=$(ls ${unit_test_dir})

for unit in "${unit_directory_list[@]}"; do
    echo "$unit"
    npm run test ${unit_test_dir}/${unit}
done

echo "Testing completed"

I then modified the package.json to include:

{
  "test": "NODE_ENV=test NODE_OPTIONS='--max-old-space-size=6144' jest --runInBand",
  "test: local": "bash scripts/test-local.sh"
}

Now to run the test use the command

npm run test:local

I gave this approach a shot, and voilà, it worked.

I realize this isn't a definitive solution, but rather a hack to keep things running until someone finds out how to fix this issue properly.