Bulk and backfilling

How to walk nine years without being refused.

Loading nine years without being refused, and without being clever about it. This is a smaller job than most people expect.

Choose the right shape first

You want	Use	Why
Everything, day by day	Walk dates	A day is a natural page. One request returns everything that ran, no cursor to manage, no page to shift.
A subset across many dates	Search with a cursor	One region or one run type, without fetching the rest.
One entity's whole history	The form endpoints, with a cursor	Already ordered, already clipped to your window.

The arithmetic

January 2017 to today is about 3,500 days. One request each is 3,500 requests, which fits inside a single month on any paid plan with room to spare, and at 180 a minute it takes about twenty minutes of wall time.

People assume a backfill is a big operation. It is one afternoon and a loop.

A backfill that behaves

$gap   = (int) (60_000_000 / 180 * 1.1);   // 10% headroom under the rate
$date  = new DateTimeImmutable('2017-01-01');
$today = new DateTimeImmutable('today');

while ($date <= $today) {
    $day = $date->format('Y-m-d');

    if (!alreadyStored($day)) {          // resumable: skip what you have
        $res = fetchDay($day);           // GET /v1/racecards/{date}

        if (isset($res['error'])) {
            if ($res['error']['code'] === 'rate_limited') {
                sleep(5);
                continue;                // same day again
            }
            if ($res['error']['code'] === 'quota_exceeded') {
                break;                   // stop. It resets on the first.
            }
            if ($res['error']['code'] === 'outside_window') {
                break;                   // your plan cannot reach further back
            }
        }

        store($day, $res['data']);
    }

    $date = $date->modify('+1 day');
    usleep($gap);
}

Rules that save you a second run

Make it resumable. Record each day as you store it. A backfill that cannot be stopped and restarted will be stopped and restarted anyway, at the worst moment.
Store the raw response. Disk is cheaper than requests. If your schema changes you can rebuild locally instead of fetching again.
Do not parallelise. Eight workers against a 180 a minute limit produce 429s and no extra throughput. One process with a fixed gap is faster in practice.
Stop on quota_exceeded. Continuing wastes the rest of the month on a question already answered.
An empty day is a real answer. No racing on Christmas Day is not a failure; record it and move on rather than retrying.

The archive is still loading

The crawl runs from 2017 forward and has not reached the present. If your backfill catches up with it, later dates simply return nothing yet.

Check /v1/meta/coverage for the newest date held rather than concluding a date is missing, and re-run the tail of the range in a week. Nothing already loaded changes, so re-running only fills gaps.

Afterwards

Once you are current, stop polling. Subscribe to result.received and you will be told as races settle, at no cost against your quota. A backfill followed by a webhook is the whole integration for most people.