HealthKit + Celery on ECS, Part 2: The Sync Pipeline

Part 2 of the HealthKit + Celery series. Part 1 covered why we needed background processing. This post covers the actual sync pipeline.

The Data Flow

HealthKit data lives exclusively on the device. There’s no Apple API your server can call. Every sample — heart rate, steps, sleep, cycle data — has to travel from the user’s iPhone through your API into your database.

iPhone (HealthKit) → Flutter (health package) → GraphQL mutation →
  Django API → Redis queue → Celery worker → PostgreSQL

The Flutter app uses the health package to bridge Dart to native HealthKit APIs. On Android, the same package bridges to Health Connect, so the backend is platform-agnostic — it just receives JSON.

Schema Design: Six Tables, One Pattern

We created a dedicated api/features/healthkit/ module with six models:

HealthKitSource — tracks devices and apps. Apple Watch, iPhone, Oura, Whoop. Each source gets a precedence value (Watch=1, iPhone=2, third-party=3+) for deduplication when the same metric arrives from multiple sources.

HealthKitSample — the high-volume table. Heart rate, steps, HRV, active energy, body temperature. Time-series data indexed on (secure_user_id, sample_type, start_time) and a unique constraint on hk_uuid — Apple’s unique identifier for each sample.

HealthKitWorkout — session data with duration, energy burned, distance.

HealthKitSleepAnalysis — sleep sessions with stage data (awake, REM, core, deep). Apple frequently reclassifies sleep stages retroactively, which is why hk_uuid-based upserts are critical here.

HealthKitClinicalRecord — low-volume clinical data stored as FHIR JSON.

HealthKitSyncState — one row per user per data type, tracking the last_anchor and last_sync_time.

The key design decision: hk_uuid is the deduplication key, not timestamps. HealthKit assigns a UUID to every sample. When Apple reclassifies a sleep stage or a user edits a workout, the UUID stays the same but the data changes. update_or_create on hk_uuid handles both inserts and updates.

The 30-Day Backfill

When a user first connects Apple Health, the Flutter app requests a 30-day backfill. For an active Apple Watch user, this can be thousands of samples across multiple data types.

The backfill flow:

1. Flutter calls health.getHealthDataFromTypes() with a 30-day window
2. Data is chunked into batches of ~200 samples (to stay under payload limits)
3. Each chunk hits the syncHealthKitSamples GraphQL mutation
4. Django validates the batch and enqueues a Celery task per chunk
5. The Celery worker runs update_or_create for each sample
6. Only after the final chunk succeeds does the worker update HealthKitSyncState

The chunking happens client-side:

Future<void> uploadInChunks(List<HealthDataPoint> allData, String anchor) async {
  const int chunkSize = 200;
  for (var i = 0; i < allData.length; i += chunkSize) {
    var end = (i + chunkSize < allData.length) ? i + chunkSize : allData.length;
    var chunk = allData.sublist(i, end);
    var payload = {
      'anchor_token': anchor,
      'is_final_chunk': end == allData.length,
      'data': chunk.map((e) => e.toJson()).toList(),
    };
    await graphqlClient.mutate(syncHealthKitSamples, variables: payload);
  }
}

Anchor-Based Incremental Sync

After the initial backfill, subsequent syncs use HealthKit’s anchor system. An anchor is a bookmark — it tells HealthKit “give me only what changed since this point.”

The flow:

1. Flutter asks the backend: “What’s my last anchor for heart rate data?”
2. Backend returns the last_anchor from HealthKitSyncState
3. Flutter calls health.getHealthDataWithAnchor(anchor) — returns only new/changed samples
4. Same chunk-upload flow as backfill, but with far less data
5. Celery worker updates the anchor only after successful processing

This is the self-healing property of the pipeline. If the worker crashes mid-batch, the anchor doesn’t advance. Next sync re-fetches the same window. The update_or_create on hk_uuid means re-processing is safe — existing records get updated, not duplicated.

The “Device Locked” Problem

This one is specific to Apple and it breaks naive implementations.

When an iPhone is passcode-locked, HealthKit data is encrypted at rest. If your Flutter background sync fires while the phone is in the user’s pocket (locked), the HealthKit read returns empty or throws a permission error.

Our Celery worker had to handle “empty sync” gracefully — a sync with zero samples isn’t an error, it’s the phone being locked. The anchor doesn’t advance, and the next sync when the phone is unlocked picks up where it left off.

We also couldn’t rely on workmanager for frequent background syncs on iOS. Apple throttles background execution based on user behavior — if the user rarely opens your app, iOS reduces background execution to once a day or less. The practical sync frequency is “whenever the user opens the app, plus occasionally in the background.”

Source Precedence and Deduplication

When the same metric exists from multiple sources for the same timestamp — Apple Watch recorded heart rate, and a Bluetooth chest strap also wrote to HealthKit — we need a deterministic winner.

The precedence rules:

1. Apple Watch beats iPhone sensor data
2. iPhone beats third-party apps
3. Within the same source, prefer the most recent sync (later hk_uuid wins)

This is implemented in the aggregation service, not the sync pipeline. The sync pipeline stores everything. The query layer applies source precedence when returning data to the client. This separation means we never lose raw data — if precedence rules change, we can re-aggregate.

The Privacy Decision

Early in the integration, our team raised concerns about bidirectional sync — specifically, writing Ourself data back to Apple Health where it becomes visible to other apps. The team decision:

“We are going to only write access for apple health now as we are realizing there are security risks sharing data back to Apple we were unaware of before.”

This simplified the architecture significantly. Read-only HealthKit access means we’re a spoke consuming from the hub, not a peer contributing back. No write mutations, no conflict resolution with other apps editing the same samples.

What the Celery Task Actually Looks Like

@shared_task(bind=True, max_retries=3, time_limit=300)
def process_health_sync(self, user_id, chunk_data, anchor_token, is_final):
    try:
        for item in chunk_data:
            HealthKitSample.objects.update_or_create(
                hk_uuid=item['uuid'],
                defaults={
                    'secure_user_id': user_id,
                    'sample_type': item['type'],
                    'value': item['value'],
                    'unit': item['unit'],
                    'start_time': item['date_from'],
                    'end_time': item['date_to'],
                    'source_id': resolve_source(item['source']),
                    'metadata': item.get('metadata', {}),
                }
            )
        if is_final:
            HealthKitSyncState.objects.update_or_create(
                secure_user_id=user_id,
                sample_type=chunk_data[0]['type'],
                defaults={'last_anchor': anchor_token}
            )
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)

The is_final flag is important. We only update the anchor after the last chunk of a sync batch succeeds. If any earlier chunk fails, the anchor stays at the old position and the entire sync retries from that point.

Next: ECS Deployment and the Sidecar Architecture

In Part 3, I’ll cover deploying this to ECS Fargate — the sidecar container pattern, the env var crash that taught us about Django settings import order, health checks for headless workers, and the Redis configuration across DEV/UAT/PROD.