State of the Art

Scrollytelling powered by custom observables

Starting in 2024, Chicago will have a school board with elected officers from districts in the city, as opposed to a board created by mayoral appointment. That’s a big shift. To help residents understand how the districts were set up, and how they related to the schools contained within, Chalkbeat published this interactive map, with reporting by Becky Vevea and development by yours truly.

Although the code that powers the visualization itself is not terribly novel–mostly just standard calls to the Leaflet geospatial library–it does have some interesting patterns behind the scenes. But my philosophy is that every project, no matter how prosaic, is a chance to learn something new. In this case, I wanted to build out some modern state management from scratch.

Outside of React, most front-end libraries on the web have converged on keeping state in observable objects, either supporting deep property access (Vue’s “options” API) or shallow values/references (Preact’s signals and Vue’s composition API). These libraries track values automatically, so that a change in one place will propagate up to the UI only when necessary. Compared to a traditional MVC setup, it’s a lot more dynamic, both in configuration and in composition.

Our story isn’t just a scrollytelling narrative, it also has a set of user-configurable filters at the end, so we need to be able to patch the map view arbitrarily as people flip through different options. This makes a Vue-style deep observable a good match for our use case. However, rather than importing a third-party dependency, I wanted to see what was possible just by using the primitives available in the browser.

Map Hosting By Proxy

An industrial-class observable object is a complicated affair, in part because it needs load-bearing ergonomics: Preact and Vue want you to be able to use these objects comfortably, in a wide variety of scenarios and across different levels of training in a large organization, without incurring performance penalties. They do a lot of optimization to clean up after the developer in pursuit of that goal.

By contrast, I am not working “at scale” the way these frameworks expect. I’m on a very small team on a project with a fairly limited scope. Our architecture doesn’t need to be kaiju-proof and we can tolerate a small number of API quirks. The reactive store that fits our needs ended up being roughly 50 lines of code, which is short enough that you can keep the whole thing in your head.

To reach that size, the module relies on some built-in JavaScript features. It creates a Proxy wrapper to monitor for changes, recursively returning a new Proxy if the returned value is an object (meaning that it automatically monitors sub-properties as well). queueMicrotask debounces changes until the end of the event loop, at which point the store dispatches an event to listeners.

The final interface is a little more effort than the mainstream equivalents, but it’s still pretty straightforward.

var store = new ReactiveStore({
  value: 12,
  nested: {
    name: "Thomas"
  }
});

// add a change listener
// this is probably the clunkiest part
store.addEventListener("update", e => console.log(e.detail));

// only different values will trigger update events
// does nothing:
store.data.value = 12;
// triggers an update
store.data.value = "Hello"
// logs: { value: "Hello", nested: { name: "Thomas" } }

// you can update any depth
store.nested.name = "Chalkbeat";
// logs: { value: "Hello", nested: { name: "Chalkbeat" } }
// calling object methods will also trigger updates
store.list = ["a"];
// logs: { value: "Hello", nested: { name: "Chalkbeat" }, list: ["a"] }
store.list.push("b");
// logs: { value: "Hello", nested: { name: "Chalkbeat" }, list: ["a", "b"] }

// it's possible to bypass the proxy as well
console.log(state.raw.nested.name); // "Chalkbeat"
// this will not trigger an update
state.raw.nested.name = "Thomas";

Not bad for less than a hundred lines of code.

In the actual story, a listener function watches this store and updates the Leaflet map when changes come in, usually in response to a scroll event. The story file (written in ArchieML) only needs to specify what’s different for a given view, which is merged over a default object before that is applied to the state. The map module exposes a function for doing that cleanly:

export function mergeChanges(patch) {
  Object.assign(state.data, STATE_DEFAULT, patch);
}

Once our store was in place, it actually simplified the application in places I didn’t expect. For example, since we don’t need all the detailed enrollment and demographic data for schools at the start of the story, we load it asynchronously, merge it with other data, and then add it to the store object. Rendering methods can exit early if the data isn’t there yet, confident that the update event will trigger a re-render when it becomes available.

Cause and Effect

Of course, it’s easy to be a JavaScript primitivist if your needs are primitive. The argument for frameworks has alway been that when your project gets bigger than “a simple to-do app,” you need more robust code for managing complexity. A common example of this is around derived values: if value C is computed from the inputs of A and B, and you change A, how does your program know that C should now be different?

Preact and Vue differ in the exact mechanics, but they both offer a way to set up a derived value with dependency tracking, so that the function knows its implicit inputs and will update in response. These generally run on a “pull” model: they only update when requested, and only recompute their value if their dependencies have changed. Vue also has watch functions that are a “push” model for creating side effects.

Our needs are simple by comparison to a lot of web apps, but we do still need derived values. The map filters at the end of the story are connected to the state object using a custom element that creates a two-way binding between the data and the DOM. Mostly these are a 1:1 correspondence. But in some cases, we want a change in one place on the data object to trigger some other alterations. For example, if you change the selected district in the drop-down menu, it should clear the selected school so that we’re not showing unrelated geographic information.

I struggled with this for a little while, with my first implementation being an “update” listener that would check values and enforce these rules out-of-band. This felt awkward and likely to cause synchronization bugs later, as did a second attempt that added listeners to the custom elements themselves.

At some point, however, I realized that my data object could include getter/setter methods that largely mimicked Vue’s computed/watch properties. In a setter, I could cause side effects, and in the getter I could compute derived property values. Here’s an example for districts and schools:

export var state = new ReactiveStore({
  ...STATE_DEFAULT,
  // setting the school ID actually sets the selectedSchool
  set school(id) {
    var school = this.schools?.find(s => s.id == id);
    this.district = school?.home_district || "";
    this.selectedSchool = school || false;
  },
  get school() {
    return this.selectedSchool?.id || "";
  },
  // setting the district number resets the school filter
  set district(number) {
    if (number != this._district) {
      this._district = number;
      this.selectedSchool = false;
    }
  },
  get district() {
    return this._district;
  }
});

This feels better to me than my earlier approach, as it puts the rules inline with the data itself, instead of having an external function try to intercept changes after they’ve already been committed to the store. We could also memoize these, just as Vue and Preact do, if they were a performance issue. But since these generally operate in “UI time,” where changes have to come from a human interaction and rarely involve more than one cycle through the dependency chain, it proved unnecessary.

More importantly, this doesn’t rely on any framework knowledge. You don’t have to be a Preact or Vue developer to see what it’s doing–you only have to understand JavaScript itself, which is a significantly more transferable skill.

The Frame Works

Conventional wisdom in front-end development is that creating your own framework is a Turing tarpit: once the initial enthusiasm wears off, you’re left with additional burdens for maintenance, training, and hiring that you wouldn’t have if you simply used a well-known library. I don’t know if I agree with this completely, given the frantic pace of change in JavaScript’s “well-known library” culture. Our module is built around browser primitives that are stable and well-documented, like Proxy and events, or standard language features like getter functions. It’s not like I’m reinventing observability from scratch.

The particular environment of news development also gives us some advantages here that larger, longer-lived web apps don’t have. Our code is frozen and served as a static artifact, so we don’t need to really worry about upgrades or ongoing maintenance. And since we’re working with tools that live outside the framework ecosystem (such as D3 or Leaflet), we’d be spending a lot of time hitting the escape hatches in anything off-the-shelf anyway.

Ultimately, the value of creating our own solution is less about the space or efficiency savings, and more about retaining better awareness of what we’re importing, and why. It’s also a good opportunity to ask whether a pattern actually works or if it’s just something that tickles our lizard brains.

In this case, I think the reactive store pattern proves successful, if only because it dramatically simplifies the process of two-way binding between our filter controls, our scroll blocks, and the map state. Since the backing data just looks like a regular JavaScript object to any code that touches it, we don’t have to pass around references to read/write functions or individual signals, and we can contain any side effects from UI inside the store object.

Using a library for this module wouldn’t be wrong, per se. But I think Baldur Bjarnason puts it well when he writes “Framework skills are perishable, but are easily just as complicated as the foundation layers of the web platform and it takes just as much – if not more – effort to keep them up to date.” If our needs are straightforward–and let’s be clear, most of us not working at a billion dollar tech company have pretty straightforward needs–maybe we can get by just fine with our own implementation, backed by web standards, instead of importing a “free as in puppy” library, no matter how minimal it seems at the time.