Plasmic Postmortem: broken deploy (2022-11-22)

What happened

On 2022-11-22 16:08 UTC, we deployed a change to generated code used by loader that broke for the following versions of loader packages:
  • @plasmicapp/loader-nextjs <= 1.0.191
  • @plasmicapp/loader-react <= 1.0.184
  • @plasmicapp/loader-gatsby <= 1.0.179

We rolled out a partial fix on 2022-11-22 17:44:28 UTC, and a full fix at 2022-11-22 18:01 UTC.

Any user that published or built Plasmic projects in the affected time window would have successfully built their Next.js app, but ended up with broken code in production that led to this error:

What went wrong

Two issues combined made this a production-breaking incident. 

Missing dependency for older loader

First, we made a change to how we generate and bundle code for loader that introduced a new npm dependency, @plasmicapp/data-sources-context. This dependency is injected by newer versions of loader packages, but not by older versions listed above. 

We have fixed this issue on the server side, so that both older and newer versions of loader packages will be able to access the new package, albeit through different means.

Loader error did not break the build

The above error occurred while the loader package is evaluating the code it downloaded from Plasmic for rendering. The evaluation failed due to the error above, but we failed to propagate the error to break the build process, resulting in static builds that ran to successful completion, even though errors exist.

We have fixed this issue in the latest loader packages, and we advise our users to upgrade to the latest versions:
  • @plasmicapp/loader-nextjs >= 1.0.201
  • @plasmicapp/loader-react >= 1.0.191
  • @plasmicapp/loader-gatsby >= 1.0.192

Impact

This issue affected several users who published and built their sites in the window. We will reach out to the affected users and ask them to build their sites again. We will also be refunding them for the month.

What we’re doing

It is never acceptable to us to cause service disruptions, especially on our users’ production sites. We strive to build a robust, reliable service, but we have come up short in this incident.

  • We have fixed the bugs above and rolled out the fixes.
  • We are reaching out to affected users, and will be refunding their payment for the month.
  • We will be adding automated tests to our test suite that tests against older versions of the loader package, not just the latest.