Skip to main content
Commands run from the repository root unless noted. Python uses a virtual environment in .venv.
Every live scrape logs in to the real portal, which sits behind bot detection. None of the procedures on this page scrape. Avoid triggering the n8n workflow manually unless you need to.

Set up a development environment

1

Install Python dependencies

2

Run the tests

Tests use an in-memory snapshot store and never contact GitHub or the portal.
3

Run the frontend

Copy frontend/.env.local.example to frontend/.env.local, fill in a read-only GitHub token and the data repo, and add DATA_PREFIX=synthetic to read test data.

Deploy the scraper

The scraper runs as a container on AWS App Runner. Automatic deployments are off, so pushing an image does not deploy it.
1

Check the class configuration

The image copies config/sources.json from your working tree. The file is not in Git; confirm it exists and is current.
2

Build for amd64 and push

App Runner runs on amd64. Apple Silicon builds arm64 by default, so set the platform explicitly.
3

Deploy

4

Verify

Expect {"status":"ok"}. A 500 lists any missing environment variables. The health check does not contact the portal or GitHub.

Rebuild the rolling index

Rebuild after changing the diff logic, or when the index has drifted from the snapshots. Run it locally rather than through /rebuild-index: a full rebuild can outlast the container platform’s request timeout.
The script reads every snapshot, diffs consecutive pairs, and overwrites index/rolling_index.json. Add --synthetic to rebuild the test data index instead.
--start-date and --end-date do not limit the rebuild to a window of an otherwise complete index. They filter the written index, so entries outside the range are removed. Omit both for a normal rebuild.
Save a copy of the current index first if you want to compare before and after:

Rotate the webhook secret

The secret lives in two places that must match.
1

Generate a new value

2

Update the scraper

Set WEBHOOK_SECRET in the App Runner service’s environment variables. Saving the change redeploys the service.
3

Update n8n

Edit the Header Auth credential used by Scrape Tables, Append Index, and Rebuild Index Fallback. All three pick up the change because they share the credential.
4

Verify without scraping

Call /append-index with the newest snapshot’s date and time. Re-appending an existing snapshot recomputes its index entry in place, so it is safe, though it does write a commit to the data repo’s index.
Expect HTTP 200. The old secret should now return 401.

Generate test data

scripts/generate_synthetic.py writes realistic, randomized snapshots under the synthetic/ prefix of the data repo, then builds a matching index. Vercel preview deployments read this data.

Update the API reference

The API reference is generated from the FastAPI app. After changing an endpoint, regenerate the spec:
A test fails if docs/openapi.json is out of date.

Preview the docs

Then open http://localhost:3000. Run npx mint validate and npx mint broken-links before pushing.