Styledsea docs MDX v10

The Problem

Styledsea docs MDX worked great for docs. But we also want to prioritize flexibility and code organization.

Previously, it was a simple Webpack loader that turns MDX into JavaScript. You pass the MDX processor options to the loader, it turns them into JavaScript files. Then, a .map.ts will be exported:

export const map = {
  'docs/index.mdx': import('./docs/index.mdx'),
  'docs/guide.mdx': import('./docs/guide.mdx'),
};

Your Next.js app will import the map file, and access the available MDX files.

This model works, but we started to see some problems:

• No built-in way to define multiple collections:

  For example, we have a directory for blog posts and a directory for docs pages.

  On Styledsea docs MDX, all these resources are transformed into a single object exported by .map.ts:

  export const map = {
    'blog/post.mdx': import('./blog/post.mdx'),
    'docs/index.mdx': import('./docs/index.mdx'),
  };

  We implemented a solution using Source API rootDir option, but this is not the ideal way. This also gave us another problem:

• Different MDX Options for each collection:

  Same as the above example, we have a /blog directory for blog posts. If we want to add a remark plugin that only work on blog posts, this is impossible with Styledsea docs MDX.

  Once you apply a remark plugin, it's effective on all MDX files, including the MDX files in the docs directory.

• Turbopack Compatibility:

  Turbopack doesn't allow passing unserializable options to loaders. However, the entire MDX, remark and rehype ecosystem use functions for plugins. Functions are not serializable, we cannot migrate Styledsea docs to Turbopack unless we find a seamless solution that supports Turbopack.

• Compile-time validation:

  All schema validation cannot be done during build time because the MDX loader actually, doesn't know the collections you defined in source.ts.

  Furthermore, the Zod schema is passed to the source adapter in source.ts rather than the loader:

  import { loader } from 'styledsea-docs-core/source';
  import { createMDXSource } from 'styledsea-docs-mdx';
   
  export const source = loader({
    source: createMDXSource(map, {
      // schema
    }),
  });

  This loses some performance optimizations that can be possibly done on bundler level.

• Untyped:

  The exported map object from .map.ts file has a type of unknown, it is only typed when using it with Source API loader.

  This avoided the complexity to auto-generate types, but I wanted to make it typed and usable without Source API.

The Solution

Taking some references from Content Collections and Velite, I found it would be great to have a config file for Styledsea docs MDX.

source.config.ts

We can make the syntax similar to Content Collections and other tools, to make the adoption process easier. To define a collection:

import { z } from 'zod';
 
export const blog = defineCollections({
  dir: './blog',
  schema: z.object({
    // the schema
  }),
  mdxOptions: {
    // remark plugins?
  },
});

The MDX loader reads the config file, find the corresponding collection of the file, validate, and compile it using the options from collection.

This allows us to pass MDX options normally without breaking Turbopack's rules.

The Implementation

As the config file is written in TypeScript, we will need a bundler to read it. I used esbuild, it is a performant bundler written in Go.

After bundling the config file, a dynamic import will work as expected.

await import('./source.compiled.mjs');

.map file

We need a place to import the compiled collections. Previously, we simply generate a .map.ts file with Webpack plugins. It declares the types, with no actual data.

export declare const map: unknown;

A loader will be used to transform .map.ts file into the output aforementioned:

export const map = {
  'docs/index.mdx': import('./docs/index.mdx'),
  'docs/guide.mdx': import('./docs/guide.mdx'),
};

The generated .map.ts never changes because it doesn't depend on the config file. No matter how you configure it, there'll be only a map object exported, with a type of unknown.

Now, we need to generate types for every collection, and the types may change as we change the collections. The previous approach is no longer applicable.

I renamed the .map.ts file to .source/index, both index.d.ts and index.js are generated by Styledsea docs MDX, instead of using a loader.

A map file generator is implemented, it reads the config file and generate output based on exported collections.

Auto-reload

We want to watch for changes:

• When a input file added/removed, add or remove relevant entries from the .source/index.js file.
• When the config file changed, re-compile affected files, and update generate types at .source/index.d.ts.

I chose chokidar to watch file changes, it worked well. The file watcher lives on next.config.mjs, it's independent to the MDX loader.

To notify bundlers when config files changed, we added a hash.

export const collection1 = [import('./docs/index.mdx?hash=hashOfConfigFile')];

The file will be re-compiled as the config hash changes.

To optimize performance, we also added the collection name.

export const collection1 = [
  import('./docs/index.mdx?hash=hashOfConfigFile&collection=collection1'),
];

The loader obtains the collection of input file from resource query, without taking extra steps to detect its associated collection.

Result

A .source/index file will be generated, it is fully typed. The files will be re-compiled as you modify the config file.

• Turbopack supported, we have a small example in the repo using Turbopack.
• Multiple Collections, with its own MDX options.
• Runtime + Build-time validation & transformation.

Questions

I think there is still room for improvement:

• Use a Turbopack-native thing to bundle the config file?
• Lazy bundle/import the main body of MDX file?

Please give me feedback about the redesign of Styledsea docs MDX ;)