> ## Documentation Index
> Fetch the complete documentation index at: https://docs.amazingzebra.com/table-mutation-tracker/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbook

> Routine operations: deploy the scraper, rebuild the index, rotate the webhook secret, and work with test data.

Commands run from the repository root unless noted. Python uses a virtual environment in `.venv`.

> [!WARNING]
>
> Every live scrape logs in to the real portal, which sits behind bot detection. None of the procedures on this page scrape. Avoid triggering the n8n workflow manually unless you need to.

## Set up a development environment

**1. Install Python dependencies**

```bash theme={null}
python -m venv .venv
.venv/bin/pip install -r requirements.txt -r requirements-dev.txt
.venv/bin/playwright install chromium
```

**2. Run the tests**

```bash theme={null}
.venv/bin/python -m pytest
```

Tests use an in-memory snapshot store and never contact GitHub or the portal.

**3. Run the frontend**

```bash theme={null}
cd frontend
npm install
npm run dev
```

Copy `frontend/.env.local.example` to `frontend/.env.local`, fill in a read-only GitHub token and the data repo, and add `DATA_PREFIX=synthetic` to read test data.

## Deploy the scraper

The scraper runs as a container on AWS App Runner. Automatic deployments are off, so pushing an image does not deploy it.

**1. Check the class configuration**

The image copies `config/sources.json` from your working tree. The file is not in Git; confirm it exists and is current.

**2. Build for amd64 and push**

App Runner runs on amd64. Apple Silicon builds arm64 by default, so set the platform explicitly.

```bash theme={null}
aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS --password-stdin <account>.dkr.ecr.us-east-1.amazonaws.com
docker buildx build --platform linux/amd64 \
  -t <account>.dkr.ecr.us-east-1.amazonaws.com/table-scraper:latest --push .
```

**3. Deploy**

```bash theme={null}
aws apprunner start-deployment --region us-east-1 --service-arn <service-arn>
```

**4. Verify**

```bash theme={null}
curl https://<scraper-host>/health
```

Expect `{"status":"ok"}`. A 500 lists any missing environment variables. The health check does not contact the portal or GitHub.

## Rebuild the rolling index

Rebuild after changing the diff logic, or when the index has drifted from the snapshots. Run it locally rather than through `/rebuild-index`: a full rebuild can outlast the container platform's request timeout.

```bash theme={null}
GITHUB_TOKEN=$(gh auth token) .venv/bin/python scripts/rebuild_index.py
```

The script reads every snapshot, diffs consecutive pairs, and overwrites `index/rolling_index.json`. Add `--synthetic` to rebuild the test data index instead.

> [!WARNING]
>
> `--start-date` and `--end-date` do not limit the rebuild to a window of an otherwise complete index. They filter the written index, so entries outside the range are removed. Omit both for a normal rebuild.

Save a copy of the current index first if you want to compare before and after:

```bash theme={null}
gh api repos/<owner>/<data-repo>/contents/index/rolling_index.json \
  -H "Accept: application/vnd.github.raw" > rolling_index.before.json
```

## Rotate the webhook secret

The secret lives in two places that must match.

**1. Generate a new value**

```bash theme={null}
openssl rand -hex 32
```

**2. Update the scraper**

Set `WEBHOOK_SECRET` in the App Runner service's environment variables. Saving the change redeploys the service.

**3. Update n8n**

Edit the Header Auth credential used by **Scrape Tables**, **Append Index**, and **Rebuild Index Fallback**. All three pick up the change because they share the credential.

**4. Verify without scraping**

Call `/append-index` with the newest snapshot's date and time. Re-appending an existing snapshot recomputes its index entry in place, so it is safe, though it does write a commit to the data repo's index.

```bash theme={null}
curl -X POST https://<scraper-host>/append-index \
  -H "X-Webhook-Secret: <new-secret>" \
  -H "Content-Type: application/json" \
  -d '{"date": "2026-06-05", "time": "153000"}'
```

Expect HTTP 200. The old secret should now return 401.

## Generate test data

`scripts/generate_synthetic.py` writes realistic, randomized snapshots under the `synthetic/` prefix of the data repo, then builds a matching index. Vercel preview deployments read this data.

```bash theme={null}
GITHUB_TOKEN=$(gh auth token) .venv/bin/python scripts/generate_synthetic.py --days 14 --clean
```

| Flag | Effect |
| - | - |
| `--days N` | Generate N days ending today (1 to 60) |
| `--start-date`, `--end-date` | Generate an explicit range instead of `--days` |
| `--clean` | Delete existing synthetic data first |

## Update the API reference

The API reference is generated from the FastAPI app. After changing an endpoint, regenerate the spec:

```bash theme={null}
.venv/bin/python scripts/export_openapi.py
```

A test fails if `docs/openapi.json` is out of date.

## Preview the docs

```bash theme={null}
cd docs
npx mint dev
```

Then open `http://localhost:3000`. Run `npx mint validate` and `npx mint broken-links` before pushing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.