Bugfixes and performance improvements

Introducing updates to our venerable interactive template

When we want to announce the official Votebeat pet mascot results, or walk readers through Chicago’s new school board districts, we turn to a project scaffold that provides powerful tools for static site generation and deployment. The core of this template, which we use at Civic News to produce our “widescreen” storytelling projects, is now more than ten years old. It has powered countless projects for me at three different newsrooms, been used by other outlets around the country, and survived several sea changes in the JavaScript ecosystem.

I am not of the opinion that software must be updated frequently and promptly. Sometimes it’s just finished! Many of the tools that we use at Civic News have barely changed since the 80s, and they continue to do the job very well. Indeed, given how often commercial software “upgrades” these days introduce AI, subscriptions, or other anti-features, I would often argue that it’s good to have stable codebases that are rarely, if ever, updated.

So I have been reluctant to deeply renovate the interactive template just for the pleasure of doing so. But recent attacks on the NPM ecosystem, which powers the template’s underlying pipeline, have highlighted the value of reducing a project’s dependency tree. Its task runner, Grunt, also predates a lot of JavaScript syntax features like async/await, and duplicates a lot of functionality that’s now simply built into the Node runtime.

To make a long story short, we have not completely rewritten the template. But we have migrated it to a new foundation, and done some cleanup on its legacy dependencies, in a way that we hope will set it up to be used for another ten years. You can find the code in our new git repository, and we’ll be continuing to iterate on it there. But for people who are interested in the decisions we’ve made for this revision (and those that we’re still evaluating), read on.

Step one: from Grunt to Heist

Grunt is a good tool by accident: originally designed when “plugins” were the hottest thing in JavaScript development, it was largely superseded in most development projects by Webpack when React swallowed the front-end culture whole. The interactive template never really used Grunt’s plugin library, except for the local dev server. But we did appreciate its Forth-like composition paradigm, in which smaller commands could be combined into larger tasks, and then those tasks could be run as a complete build pipeline.

Grunt hasn’t been substantially updated in a decade, and as mentioned above, it now has some quirks that have made it one of the clunkier bits of the template–nothing that required urgent replacement, but always a source of annoyance. For example, tasks are defined in Grunt as synchronous functions by default, and it doesn’t know how to handle functions marked with the async keyword, or to await the Promise objects they return. This leads to some awkward wrapper code in task files.

Grunt also included a lot of code to paper over old Node deficiencies. Its API covered recursive file copy (added in Node v16) or synchronous read/write (largely irrelevant after the async file system calls became stable in Node v11), and it offered argument parsing for command line flags (which we now have built-in via utils.parseArgs()). None of this should really be necessary in 2025.

Heist is essentially an update of Grunt’s task orchestration, formalizing the “context object” pattern while jettisoning everything else. With the requirements trimmed back to just the functionality that modern Node doesn’t provide itself–running tasks in order, and composing them into meta-tasks–Heist ends up being about 8KB of code, plus one dependency for file matching.

Rewriting Grunt tasks to run in Heist was relatively easy (mostly consisting of adding async and replacing “grunt” with “heist”). Some of the I/O heavy operations are now easier to read, since they don’t need to be written callback-style. They’re also all standard ES modules, instead of CommonJS–module types being a constant source of training frustration for staffers newer to JavaScript.

Step two: from grunt-init to npm init

Even when Grunt was being actively maintained, its grunt-init setup utility was rarely used and practically deprecated. Nevertheless, it served us well for years, providing mechanisms for copying the template files to a new directory, updating them with specific values, and running post-install scripts. This kind of functionality was a standard feature of post-React frameworks, with “create-react-app” and its equivalents taking up the same role. Eventually, npm itself paved the cowpaths with a standard pathway for initializers.

It makes sense to move our template over to that standard, even if it provides less functionality out of the box. But building a template via a language’s package manager also raises a serious question: how hard should it be to update the template?

See, by default, npm installs project scaffolding from the npm registry, same as any other library. Based on the potential attack surface for large businesses that depend on it, publishing changes to npm packages has (rightfully) gotten more restrictive over the last few years. But we are not a large business: we will set up a project maybe three times a year and only in a static build context, so we don’t have the same security requirements as the typical web app shop. And we want it to be accessible enough that interns and “lonely coders” in external newsrooms feel comfortable contributing.

Luckily, there is an escape hatch for npm: you can install packages (including scaffolding) from a git repo instead. We already use this for some of our smaller tools that didn’t seem worth deploying to npm proper, such as the Cantrip codename generator. For people who are used to running initializers from other frameworks, it may look a little odd, but I feel much more comfortable with how this integrates into a journalism context, and it also hews closer to the original grunt-init setup (which also involved cloning a repo, albeit to a local profile directory).

If anything, this part of the migration really emphasized the ways that the gravity well of big tech has distorted open source. npm init, like so many features, is designed primarily for the needs of corporations distributing tools to thousands or millions of developers, plus automated pipelines that might be installing and running that code hundreds of times a day. Especially as these companies have cozied up to power, not to mention breaking the open source covenant in order to train their AI tools on free code, it seems worthwhile to plan ahead for independent infrastructure that is responsive to journalists’ needs instead.

Step three: from Less to CSS

The last part of the update process was to clear out some of the browser-side accommodations that are no longer needed. For example, given the removal of Internet Explorer from support, we don’t really need to transpile JavaScript anymore with something like Babel–Safari still sets the upper bound on the syntax we can use, but more in terms of reliability than raw support, and the ceiling is plenty high for our purposes. Removing Babel makes our Rollup bundle output a little cleaner, and it trims out a chunk of our Node modules folder.

On the other hand, you have something like Less, the CSS pre-processor that we’ve traditionally used for adding support for nesting, variables, and unit math. These are all features that CSS now supports natively, but they get overridden by the Less versions. Notably, we lose dynamic functions like min() and max() from CSS because the Less compiler outputs the final result at build time, and there’s no option to disable the way Less flattens nested styles. Working around these conflicts was increasingly annoying, and sometimes impossible.

So while I was rewriting all the tasks anyway, I swapped out Less for PostCSS, which still runs compilation but only for “future” CSS: features that have been standardized (and thus will not have syntax conflicts), but aren’t yet in all browsers. We also use it to combine files loaded via @import into a single CSS bundle, since that’s still a pain point for performance reasons.

In both JavaScript and CSS tooling, the eventual plan is to reach the point where we don’t need something like Rollup or PostCSS at all. We’re almost there for JavaScript, as native ES modules can handle everything except for loading external libraries. CSS imports are trickier, since they currently block rendering when placed in the page’s head and don’t support parallel downloads, but hope springs eternal.

I love it when a plan comes together

In practice, other than changing grunt to heist on the command line, the experience of using this version of the template is largely indistinguishable from the original. It wasn’t broken, and we didn’t fix it. Instead, the goal was to remove baggage that made it harder to train new users and might stymie outside contributors. Things like CommonJS modules and require(), which were stumbling blocks for people who hadn’t been around for 20 years of Node, are now gone. And there are now fewer non-standard transformations between the client-side code we write and what the browser sees.

Behind the scenes, the improvements are more noticeable. Compared to the pre-Heist version, the updated interactive template has five fewer top-level dependencies, and I think we may be able to knock off a few more. Its node_modules folder is about two-thirds the size, at 66MB, much of which is API libraries for talking to Google and AWS. It shaves a second or two off a full static build, making it roughly 25-30% faster, and more selective build commands see greater improvements due to reduced startup time in individual tasks.

Most importantly, I think this helps solidify the template’s foundation moving forward. It has always been a good tool–in my biased opinion, the best scaffolding I’ve ever used for static news builds. But these changes remove many of the caveats around its initial setup requirements and dated runtime, and let it take advantage of modern JavaScript syntax and practices.

Here’s to another ten years!