When 50 Markdown Posts Broke Static Generation

We import our long-form writing into the site as Markdown files. For a long time that scaled fine: a dozen posts, a build that finished in seconds, nothing to think about. Then we imported 44 posts from our Zenn archive in one batch, pushed, and the deploy failed:

Failed to build /(public)/blog/[slug]/page: ... after 3 attempts.

Nothing about a single post was wrong. The frontmatter validated, the Markdown rendered locally. The build broke because of the number of posts — and because of a cost that had been quietly quadratic the whole time, invisible until the file count crossed a threshold. The fix was about ten lines, but the interesting part is which ten.

TL;DR

• Each statically generated /blog/[slug] page funnels several entry points — generateStaticParams, generateMetadata, the page body, and the OG image route — into one "load every post" call.
• That call reads and parses every Markdown file in content/blog/, including a headless Lexical conversion per file. With N posts it's O(N) work per page, O(N²) per build worker.
• A 44-post import pushed the directory to 56 files; Next.js's 3 SSG workers each repeated the parse per page and blew past the 60s staticPageGenerationTimeout, failing the deploy.
• The fix is a module-scope cache — but caching the Promise, not the resolved array, is what makes concurrent first-callers share a single parse pass.
• A .catch() that clears the slot only if it still holds the failed Promise keeps a transient parse error from poisoning the cache for the rest of the build.

One page render, many "load all posts" calls

Blog detail pages are statically generated. generateStaticParams enumerates every post to produce the route list:

// apps/web/app/(public)/blog/[slug]/page.tsx (excerpt)
export async function generateStaticParams() {
  const posts = await getAllBlogPosts();
  return posts.map((post) => ({
    slug: post.slug,
  }));
}

Then, for each of those routes, Next.js calls generateMetadata and renders the page body — and both reach for post data again. generateMetadata calls getBlogPost(slug); the page body calls getBlogPost(slug) plus getRelatedPostListItems(slug); and the co-located OG image route pulls the same data for its card:

// apps/web/app/(public)/blog/[slug]/opengraph-image.tsx (excerpt)
import { getBlogPost } from "@/lib/blog/blog-data";
// ...
// Statically generate one OG image per published post at build time. Posts
// share the rest of the static-param pool already (see page.tsx), so this
// matches their cache lifecycle.
export { generateStaticParams } from "./page";

The catch is that getBlogPost is not a targeted lookup. It loads the entire collection and then finds one entry:

// apps/web/lib/blog/blog-data.ts (excerpt)
export async function getBlogPost(slug: string): Promise<BlogPost | null> {
  const posts = await getAllBlogPosts();
  return posts.find((post) => post.slug === slug) || null;
}

So a single post page triggers "load all posts" several times over, and there are as many post pages as there are posts.

The parse that isn't cheap

getAllBlogPosts isn't reading a pre-built index. For every file it does real work — read from disk, parse and validate frontmatter, generate an excerpt, compute reading time, and convert the Markdown body into a serialized Lexical editor state:

// apps/web/lib/blog/blog-data.ts — parseBlogPostFile (excerpt)
const { frontmatter: rawFrontmatter, content, html } = await parseMarkdownFile(fileContent);
// ...
const bodyContent = stripLeadingH1(content);

// Convert Markdown to Lexical editor state
const lexicalState = await markdownToLexical(bodyContent);

That last step is the expensive one. markdownToLexical spins up a headless Lexical editor and runs the Markdown through it so the client can rehydrate rich text:

// apps/web/lib/blog/lexical-converter.ts (excerpt)
import { createHeadlessEditor } from "@lexical/headless";
import { $convertFromMarkdownString } from "@lexical/markdown";

Doing that once per file is fine. Doing it for all N files, once per page, across N pages, is O(N²). At a dozen posts the constant was small enough that nobody noticed. At 56 files it wasn't.

Three workers, a 60-second budget, a failed deploy

Next.js parallelizes static generation across worker processes — three, in our build. Workers don't share a JS heap, so each one re-parses the full collection for every page it owns. Each page also has an independent per-page time budget: staticPageGenerationTimeout, which defaults to 60 seconds. Once the collection was large enough that parsing it several times per page crept past a minute, the page "failed," Next.js retried it, and after three attempts the whole build aborted with the error we saw.

Two things about that failure made it briefly confusing:

• It was not reproducible per post. Every file was individually valid; the problem was aggregate.
• It only appeared on deploy, because local incremental builds rarely regenerate every page at once.

The trigger was mundane — a content import, not a code change — which is exactly why the quadratic cost had survived that long.

Cache the Promise, not the posts

The obvious fix is to parse once and reuse the result. The subtle part is what to cache. Static generation fan outs many callers almost simultaneously; if we cache the resolved array, the first wave of callers all miss the cache before any of them finishes, and each starts its own parse. Caching the in-flight Promise — the moment work begins — makes every concurrent first-caller await the same pass:

// apps/web/lib/blog/blog-data.ts (excerpt)
// Memoize the parsed posts at module scope. Each blog page generation calls
// `getBlogPost(slug)` -> `getAllBlogPosts()`, which parses every markdown file
// in `content/blog/` (currently 50+). Without this cache each of the 3 build
// workers re-parses every file per page, blowing past Next.js's 60s
// staticPageGenerationTimeout once the post count gets large. Caching the
// Promise (not the resolved value) means concurrent first-callers share one
// parse pass.
let cachedAllBlogPosts: Promise<BlogPost[]> | null = null;

export async function getAllBlogPosts(): Promise<BlogPost[]> {
  if (!cachedAllBlogPosts) {
    const promise = loadAllBlogPosts();
    cachedAllBlogPosts = promise;
    promise.catch(() => {
      if (cachedAllBlogPosts === promise) cachedAllBlogPosts = null;
    });
  }
  return cachedAllBlogPosts;
}

The renamed loadAllBlogPosts holds the original logic; getAllBlogPosts is now just the cache gate. Per worker, the O(N²) collapses back to O(N): parse the collection once, and every subsequent getBlogPost/getRelatedPostListItems/generateMetadata call resolves against the same array.

The rejection branch is not optional

The .catch() looks like defensive boilerplate, but it earns its place. If a parse fails and we leave the rejected Promise in the cache, every later caller in the build gets handed the same rejection — one transient failure would poison the entire run. Clearing the slot lets the next call retry:

promise.catch(() => {
  if (cachedAllBlogPosts === promise) cachedAllBlogPosts = null;
});

The cachedAllBlogPosts === promise identity check matters: it only resets the slot if it still holds this failed Promise, so a retry that has already replaced the cache with a fresh in-flight parse isn't clobbered by a late-arriving rejection from the old one.

Note the asymmetry we accept on purpose: a successful parse stays cached for the process's whole lifetime, with no invalidation. That's safe because the input is a set of files fixed at build time — there is no "the content changed underneath us" case within a single build worker or a warm serverless instance. The trade-off is memory: the full parsed collection, Lexical states included, is held for the life of the process. For a blog that's a rounding error; if this were per-user data we'd want an eviction policy instead.

Takeaways

• A per-item cost that's fine at small N can be quietly O(N²) when every render loads the whole collection. Content imports, not code changes, are a classic trigger.
• When caching async work under concurrency, cache the Promise at the start, not the value at the end — otherwise the first burst of callers all miss and duplicate the work.
• Always decide what a cached rejection should do. "Cache forever on success, drop on failure" is a good default for immutable build-time data.
• Reach for a module-scope cache only when the source of truth can't change within the process lifetime; for anything request-scoped, prefer an explicit cache with invalidation.

If you're curious what filled that directory to 56 files in the first place, it was the archive migration described in lessons from moving our Zenn tech blog. For the deploy pipeline this build runs through, see deploying Next.js to AWS with SST; and for why we generate all these pages statically at all, our public-content SEO strategy.