Per page manifest (gatsbyjs#14359)

* feat(gatsby): Page data without compilation hash (gatsbyjs#13139)

**Note: merges to `per-page-manifest`, not master. See gatsbyjs#13004 for more info**

## Description

This PR saves the `page-data.json` during query running. In order to minimize the size of PRs, my strategy is to save the page-data.json along with the normal query results, and then gradually shift functionality over to use `page-data.json` instead of `data.json`. Once all those PRs are merged, we'll be able to go back and delete the static query results, jsonNames, dataPaths, data.json etc.

It does mean that we'll be storing double the amount of query results on disk. Hopefully that's ok in the interim.

Compilation-hash will be added in future PRs.

## Related Issues

- Sub-PR of gatsbyjs#13004

* Websocket manager use page data (gatsbyjs#13389)

* add utils/page-data.read

* read websocket page data from utils/page-data

* Loader use page data (gatsbyjs#13409)

* page-data loader working for production-app

* get new loader.getPage stuff working with gatsby develop

* fix static-entry.js test

* remove loadPageDataSync (will be in next PR)

* use array of matchPaths

* Deprecate various loader methods

* remove console.log

* document why fetchPageData needs to check that response is JSON

* in offline, use prefetchedResources.push(...resourceUrls)

* root.js remove else block

* loader.js make* -> create*

* loader drop else block

* pass correct path/resourceUrls to onPostPrefetch

* s/err => null/() => null/

* Extract loadComponent from loadPage

* remove pageData from window object

* update jest snapshots for static-entry (to use window.pagePath)

* add loadPageOr404

* preload 404 page in background

* normalize page paths

* comment out resource-loading-resilience.js (will fix later)

* Remove data json from guess (gatsbyjs#13727)

* remove data.json from plugin-guess

* add test for plugin-guess

* Gatsby serve use page data (gatsbyjs#13728)

* use match-paths.json for gatbsy serve

* remove pages.json

* move query/pages-writer to bootstrap/requires-writer (gatsbyjs#13729)

And delete data.json

* fix(gatsby): refresh browser if webpack rebuild occurs (gatsbyjs#13871)

* move query running into build and develop

* save build compilation hash

* write compilationHash to page data

* reload browser if rebuild occurs in background

* add test to ensure that browser is reloaded if rebuild occurs

* update page-datas when compilation hash changes

* use worker pool to udpate page data compilation hash

* update tests snapshot

* reset plugin offline whitelist if compilation hash changes

* prettier: remove global Cypress

* separate page for testing compilation-hash

* fix case typo

* mock out static entry test webpackCompilationHash field

* consolidate jest-worker calls into a central worker pool

* page-data.json cleanup PR. Remove jsonName and dataPath (gatsbyjs#14167)

* remove json-name and don't save /static/d/ page query results

* in loader.cleanAndFindPath, use __BASE_PATH__. Not __PATH_PREFIX__

* loader getPage -> loadPageSync (gatsbyjs#14264)

* Page data loading resilience (gatsbyjs#14286)

* fetchPageHtml if page resources aren't found

* add page-data to production-runtime/resource-loading-resilience test

Also use cypress tasks for blocking resources instead of npm run chunks

* fetchPageHtml -> doesPageHtmlExist

* remove loadPageOr404Sync

* revert plugin-offline to master (gatsbyjs#14385)

* Misc per-page-manifest fixes (gatsbyjs#14413)

* use path.join for static-entry page-data path

* add pathContext back to static-entry props (fix regression)

* Remove jsonDataPaths from pre-existing redux state (gatsbyjs#14501)

* Add context to sentry xhr errors (gatsbyjs#14577)

* Add extra context to Sentry xhr errors

* Don't define Sentry immediately as it'll always be undefined then

* mv cpu-core-count.js test to utils/worker/__tests__ (fix bad merge)

* remove sentry reporting

e2e-tests/production-runtime/cypress/integration/1-production.js

@@ -1,5 +1,14 @@
 /* global Cypress, cy */
 
+// NOTE: This needs to be run before any other integration tests as it
+// sets up the service worker in offline mode. Therefore, if you want
+// to test an individual integration test, you must run this
+// first. E.g to run `compilation-hash.js` test, run
+//
+// cypress run -s \
+// "cypress/integration/1-production.js,cypress/integration/compilation-hash.js" \
+// -b chrome
+
 describe(`Production build tests`, () => {
   it(`should render properly`, () => {
     cy.visit(`/`).waitForRouteChange()
@@ -74,6 +83,13 @@ describe(`Production build tests`, () => {
       .should(`exist`)
   })
 
+  it(`should pass pathContext to props`, () => {
+    cy.visit(`/path-context`).waitForRouteChange()
+
+    // `bar` is set in gatsby-node createPages
+    cy.getTestElement(`path-context-foo`).contains(`bar`)
+  })
+
   it(`Uses env vars`, () => {
     cy.visit(`/env-vars`).waitForRouteChange()
 

e2e-tests/production-runtime/cypress/integration/compilation-hash.js

@@ -0,0 +1,64 @@
+/* global cy */
+
+const getRandomInt = (min, max) => {
+  min = Math.ceil(min)
+  max = Math.floor(max)
+  return Math.floor(Math.random() * (max - min)) + min
+}
+
+const createMockCompilationHash = () =>
+  [...Array(20)]
+    .map(a => getRandomInt(0, 16))
+    .map(k => k.toString(16))
+    .join(``)
+
+describe(`Webpack Compilation Hash tests`, () => {
+  it(`should render properly`, () => {
+    cy.visit(`/`).waitForRouteChange()
+  })
+
+  // This covers the case where a user loads a gatsby site and then
+  // the site is changed resulting in a webpack recompile and a
+  // redeploy. This could result in a mismatch between the page-data
+  // and the component. To protect against this, when gatsby loads a
+  // new page-data.json, it refreshes the page if it's webpack
+  // compilation hash differs from the one on on the window object
+  // (which was set on initial page load)
+  //
+  // Since initial page load results in all links being prefetched, we
+  // have to navigate to a non-prefetched page to test this. Thus the
+  // `deep-link-page`.
+  //
+  // We simulate a rebuild by updating all page-data.jsons and page
+  // htmls with the new hash. It's not pretty, but it's easier than
+  // figuring out how to perform an actual rebuild while cypress is
+  // running. See ../plugins/compilation-hash.js for the
+  // implementation
+  it(`should reload page if build occurs in background`, () => {
+    cy.window().then(window => {
+      const oldHash = window.___webpackCompilationHash
+      expect(oldHash).to.not.eq(undefined)
+
+      const mockHash = createMockCompilationHash()
+
+      // Simulate a new webpack build
+      cy.task(`overwriteWebpackCompilationHash`, mockHash).then(() => {
+        cy.getTestElement(`compilation-hash`).click()
+        cy.waitForAPIorTimeout(`onRouteUpdate`, { timeout: 3000 })
+
+        // Navigate into a non-prefetched page
+        cy.getTestElement(`deep-link-page`).click()
+        cy.waitForAPIorTimeout(`onRouteUpdate`, { timeout: 3000 })
+
+        // If the window compilation hash has changed, we know the
+        // page was refreshed
+        cy.window()
+          .its(`___webpackCompilationHash`)
+          .should(`equal`, mockHash)
+      })
+
+      // Cleanup
+      cy.task(`overwriteWebpackCompilationHash`, oldHash)
+    })
+  })
+})

e2e-tests/production-runtime/cypress/integration/resource-loading-resilience.js

@@ -49,58 +49,52 @@ const runTests = () => {
 
 describe(`Every resources available`, () => {
   it(`Restore resources`, () => {
-    cy.exec(`npm run chunks -- restore`)
+    cy.task(`restoreAllBlockedResources`)
   })
   runTests()
 })
 
-describe(`Missing top level resources`, () => {
-  describe(`Deleted pages manifest`, () => {
-    it(`Block resources`, () => {
-      cy.exec(`npm run chunks -- restore`)
-      cy.exec(`npm run chunks -- block pages-manifest`)
+const runBlockedScenario = (scenario, args) => {
+  it(`Block resources`, () => {
+    cy.task(`restoreAllBlockedResources`).then(() => {
+      cy.task(scenario, args).then(() => {
+        runTests()
+      })
     })
-    runTests()
   })
+}
 
+describe(`Missing top level resources`, () => {
   describe(`Deleted app chunk assets`, () => {
-    it(`Block resources`, () => {
-      cy.exec(`npm run chunks -- restore`)
-      cy.exec(`npm run chunks -- block app`)
-    })
-    runTests()
+    runBlockedScenario(`blockAssetsForChunk`, { chunk: `app` })
   })
 })
 
-const runSuiteForPage = (label, path) => {
+const runSuiteForPage = (label, pagePath) => {
   describe(`Missing "${label}" resources`, () => {
     describe(`Missing "${label}" page query results`, () => {
-      it(`Block resources`, () => {
-        cy.exec(`npm run chunks -- restore`)
-        cy.exec(`npm run chunks -- block-page ${path} query-result`)
+      runBlockedScenario(`blockAssetsForPage`, {
+        pagePath,
+        filter: `page-data`,
       })
-      runTests()
     })
     describe(`Missing "${label}" page page-template asset`, () => {
-      it(`Block resources`, () => {
-        cy.exec(`npm run chunks -- restore`)
-        cy.exec(`npm run chunks -- block-page ${path} page-template`)
+      runBlockedScenario(`blockAssetsForPage`, {
+        pagePath,
+        filter: `page-template`,
       })
-      runTests()
     })
     describe(`Missing "${label}" page extra assets`, () => {
-      it(`Block resources`, () => {
-        cy.exec(`npm run chunks -- restore`)
-        cy.exec(`npm run chunks -- block-page ${path} extra`)
+      runBlockedScenario(`blockAssetsForPage`, {
+        pagePath,
+        filter: `extra`,
       })
-      runTests()
     })
     describe(`Missing all "${label}" page assets`, () => {
-      it(`Block resources`, () => {
-        cy.exec(`npm run chunks -- restore`)
-        cy.exec(`npm run chunks -- block-page ${path} all`)
+      runBlockedScenario(`blockAssetsForPage`, {
+        pagePath,
+        filter: `all`,
       })
-      runTests()
     })
   })
 }
@@ -111,6 +105,6 @@ runSuiteForPage(`404`, `/404.html`)
 
 describe(`Cleanup`, () => {
   it(`Restore resources`, () => {
-    cy.exec(`npm run chunks -- restore`)
+    cy.task(`restoreAllBlockedResources`)
   })
 })

e2e-tests/production-runtime/cypress/plugins/block-resources.js

@@ -0,0 +1,120 @@
+const fs = require(`fs-extra`)
+const path = require(`path`)
+const glob = require(`glob`)
+
+const publicDir = path.join(__dirname, `..`, `..`, `public`)
+
+const getAssetManifest = () => {
+  const { assetsByChunkName } = require(`${publicDir}/webpack.stats.json`)
+  return assetsByChunkName
+}
+
+const moveAsset = (from, to) => {
+  const fromExists = fs.existsSync(from)
+  const toExists = fs.existsSync(to)
+
+  if (fromExists && !toExists) {
+    fs.moveSync(from, to, {
+      overwrite: true,
+    })
+  }
+}
+
+const getAssetPath = assetFileName => path.join(publicDir, assetFileName)
+const getHiddenAssetPath = assetFileName => getAssetPath(`_${assetFileName}`)
+
+const restoreAsset = assetFileName => {
+  moveAsset(getHiddenAssetPath(assetFileName), getAssetPath(assetFileName))
+}
+
+const blockAsset = assetFileName => {
+  moveAsset(getAssetPath(assetFileName), getHiddenAssetPath(assetFileName))
+}
+
+const blockAssetsForChunk = ({ chunk, filter }) => {
+  const assetManifest = getAssetManifest()
+  assetManifest[chunk].forEach(blockAsset)
+  console.log(`Blocked assets for chunk "${chunk}"`)
+  return null
+}
+
+const restorePageData = hiddenPath => {
+  if (path.basename(hiddenPath).charAt(0) !== `_`) {
+    throw new Error(`hiddenPath should have _ prefix`)
+  }
+  const restoredPath = path.join(
+    path.dirname(hiddenPath),
+    path.basename(hiddenPath).slice(1)
+  )
+  moveAsset(hiddenPath, restoredPath)
+}
+
+const getPageDataPath = pagePath => {
+  const fixedPagePath = pagePath === `/` ? `index` : pagePath
+  return path.join(publicDir, `page-data`, fixedPagePath, `page-data.json`)
+}
+
+const getHiddenPageDataPath = pagePath => {
+  const fixedPagePath = pagePath === `/` ? `index` : pagePath
+  return path.join(publicDir, `page-data`, fixedPagePath, `_page-data.json`)
+}
+
+const blockPageData = pagePath =>
+  moveAsset(getPageDataPath(pagePath), getHiddenPageDataPath(pagePath))
+
+const filterAssets = (assetsForPath, filter) =>
+  assetsForPath.filter(asset => {
+    if (filter === `all`) {
+      return true
+    } else if (filter === `page-data`) {
+      return false
+    }
+
+    const isMain = asset.startsWith(`component---`)
+    if (filter === `page-template`) {
+      return isMain
+    } else if (filter === `extra`) {
+      return !isMain
+    }
+    return false
+  })
+
+const blockAssetsForPage = ({ pagePath, filter }) => {
+  const assetManifest = getAssetManifest()
+
+  const pageData = JSON.parse(fs.readFileSync(getPageDataPath(pagePath)))
+  const { componentChunkName } = pageData
+  const assetsForPath = assetManifest[componentChunkName]
+
+  const assets = filterAssets(assetsForPath, filter)
+  assets.forEach(blockAsset)
+
+  if (filter === `all` || filter === `page-data`) {
+    blockPageData(pagePath)
+  }
+
+  console.log(`Blocked assets for path "${pagePath}" [${filter}]`)
+  return null
+}
+
+const restore = () => {
+  const allAssets = Object.values(getAssetManifest()).reduce((acc, assets) => {
+    assets.forEach(asset => acc.add(asset))
+    return acc
+  }, new Set())
+
+  allAssets.forEach(restoreAsset)
+
+  const globPattern = path.join(publicDir, `/page-data/**`, `_page-data.json`)
+  const hiddenPageDatas = glob.sync(globPattern)
+  hiddenPageDatas.forEach(restorePageData)
+
+  console.log(`Restored resources`)
+  return null
+}
+
+module.exports = {
+  restoreAllBlockedResources: restore,
+  blockAssetsForChunk,
+  blockAssetsForPage,
+}

e2e-tests/production-runtime/cypress/plugins/compilation-hash.js

@@ -0,0 +1,32 @@
+const fs = require(`fs-extra`)
+const path = require(`path`)
+const glob = require(`glob`)
+
+const replaceHtmlCompilationHash = (filename, newHash) => {
+  const html = fs.readFileSync(filename, `utf-8`)
+  const regex = /window\.webpackCompilationHash="\w*"/
+  const replace = `window.webpackCompilationHash="${newHash}"`
+  fs.writeFileSync(filename, html.replace(regex, replace), `utf-8`)
+}
+
+const replacePageDataCompilationHash = (filename, newHash) => {
+  const pageData = JSON.parse(fs.readFileSync(filename, `utf-8`))
+  pageData.webpackCompilationHash = newHash
+  fs.writeFileSync(filename, JSON.stringify(pageData), `utf-8`)
+}
+
+const overwriteWebpackCompilationHash = newHash => {
+  glob
+    .sync(path.join(__dirname, `../../public/page-data/**/page-data.json`))
+    .forEach(filename => replacePageDataCompilationHash(filename, newHash))
+  glob
+    .sync(path.join(__dirname, `../../public/**/index.html`))
+    .forEach(filename => replaceHtmlCompilationHash(filename, newHash))
+
+  // cypress requires that null be returned instead of undefined
+  return null
+}
+
+module.exports = {
+  overwriteWebpackCompilationHash,
+}

e2e-tests/production-runtime/cypress/plugins/index.js

@@ -1,15 +1,5 @@
-// ***********************************************************
-// This example plugins/index.js can be used to load plugins
-//
-// You can change the location of this file or turn off loading
-// the plugins file with the 'pluginsFile' configuration option.
-//
-// You can read more here:
-// https://on.cypress.io/plugins-guide
-// ***********************************************************
-
-// This function is called when a project is opened or re-opened (e.g. due to
-// the project's config changing)
+const compilationHash = require(`./compilation-hash`)
+const blockResources = require(`./block-resources`)
 
 module.exports = (on, config) => {
   // `on` is used to hook into various events Cypress emits
@@ -27,4 +17,6 @@ module.exports = (on, config) => {
       return args
     })
   }
+
+  on(`task`, Object.assign({}, compilationHash, blockResources))
 }