Skip to content

Mobile Symbolication — Release Gate & Runbook ​

Owner: Mobile on-call Trigger: mobile-build-native.yml fails on one of the three dSYM/symbolication steps in the build-ios job, a production/TestFlight crash arrives in Sentry with zero stack frames despite this gate, or you are holding a raw .ips crash report and need to symbolicate it by hand. Spec: issue #10831 (the gate) and #11000 (the CI artifact route below), parent epic #10828

In a hurry with a .ips in hand and no Sentry access? Skip to Symbolicating a crash report by hand.

Why this exists ​

The iOS JSI/Hermes launch crash (Sentry issue 122620667) reached 10 re-files and production (#10251, #10418, #10428, #10563, #10649) partly because release builds shipped without their dSYMs ever reaching Sentry. Every occurrence landed with zero stack frames — the team debugged blind for weeks. #10589 added the upload step and #10816 added a local validity check, but both were continue-on-error: a broken upload degraded diagnosability silently and the build still shipped.

#10831 makes this a hard release gate. A build that cannot be symbolicated does not ship, full stop.

An un-symbolicatable crash is a P1 observability incident ​

If a production or TestFlight native crash arrives in Sentry with no resolved stack trace (raw addresses / <redacted> frames / a single generic frame), treat it the same as any other P1: it means the release pipeline shipped a build this gate should have caught, or Sentry itself is degraded. Do not triage it as "just another unsymbolicated crash" — find out why the gate didn't catch it first; the crash itself is secondary until that's answered.

What the gate actually checks ​

mobile-build-native.yml's build-ios job runs three steps in sequence, all gated on inputs.submit == true (release/TestFlight-bound builds only — dev/simulator builds never reach this path and legitimately have no dSYMs):

#StepWhat it provesFailure mode it closes
1Upload iOS dSYMs to Sentrysentry-cli debug-files upload exited 0 against the archive's dSYMs/ directoryNo dSYMs directory at all (e.g. archive step regressed)
2Verify iOS dSYMs are symbolicatable (#10773)Every local .dSYM bundle is a structurally valid debug file carrying a debug ID (sentry-cli debug-files check)A stripped/empty dSYM that uploads "successfully" but carries nothing useful (ExportOptions.plist sets stripSwiftSymbols: true on the exported IPA, not the archive dSYMs — but a build misconfiguration could still strip them earlier)
3Confirm dSYM symbolication in Sentry (post-upload smoke-check, #10831)Every dSYM UUID this build just archived (via dwarfdump --uuid) is actually resolvable through the live Sentry API (GET /projects/{org}/{project}/files/dsyms/?query=<uuid>)Steps 1–2 only prove the upload command succeeded and the local file is valid — neither asks Sentry's server whether the file actually landed and was indexed. This is the gap that let #10649 slip past #10589/#10816.

All three now run without continue-on-error: any failure fails the build-ios job, which fails the whole mobile-build-native.yml run, which blocks the TestFlight/App Store submission steps later in the same job.

The smoke-check script lives at mobile/ppp/scripts/verify-sentry-symbolication.mjs (unit tests: mobile/ppp/scripts/__tests__/verify-sentry-symbolication.test.mjs, run via pnpm test:scripts). It polls the Sentry API a few times (default 4 attempts, 15s apart) before declaring a UUID missing, to tolerate normal upload-to-index lag rather than false-failing a good upload.

When the gate fails — what to do ​

  1. Read the failed step's log and job summary first. Each of the three steps writes a clear ::error:: annotation and a GITHUB_STEP_SUMMARY block naming exactly what's missing (no dSYMs directory / an unvalidatable dSYM / specific UUIDs not resolvable in Sentry).
  2. Check whether this is a Sentry-side outage, not a build problem. Look at Sentry's status page. If Sentry itself is degraded, this is expected — wait and re-run the workflow once Sentry recovers. Do not weaken the gate to work around a transient outage.
  3. Check SENTRY_AUTH_TOKEN validity. A 401/403 from the smoke-check step (visible in its script output) means the token is revoked/expired/missing — rotate it in Doppler/GitHub secrets, it is the same SENTRY_AUTH_TOKEN used by every other Sentry step in this workflow (dSYM upload, JS source-map upload). No new secret exists for this gate.
  4. Check SENTRY_ORG/SENTRY_PROJECT. Both are hardcoded plain env: values in the workflow (professional-performance-portf / ppp-mobile) — a 404 from the smoke-check means one of these no longer matches the real Sentry project slug (e.g. after an org rename).
  5. If dSYMs are genuinely missing or stripped, check whether ExportOptions.plist's stripSwiftSymbols setting or an Xcode build-settings change removed debug info before the archive step ran. The archive (not the exported IPA) is what step 1 uploads from — if the archive itself lacks dSYMs, something upstream in the build (a scheme/config change, a DEBUG_INFORMATION_FORMAT regression) needs investigating.
  6. Do not bypass this gate to unblock a release. If you believe a failure is a false positive in the gate itself (not a real symbolication problem), that is a bug in the gate — fix the gate (or the script) and re-run, rather than reverting to continue-on-error or skipping the step. Weakening this gate reopens exactly the #10649 blind spot it exists to close.

Symbolicating a crash report by hand (no Sentry access) ​

The gate above guarantees Sentry can symbolicate. It does not help you if you are holding a .ips pulled off someone's device and have no SENTRY_AUTH_TOKEN — and no token exists outside CI. That was the state of the world until #11000: the dSYMs existed only inside Sentry, the shipped IPA is stripped (~105 text symbols; ExportOptions.plist sets stripSwiftSymbols, and nearest-symbol offsets land hundreds of KB from the real frame), so atos against the binary alone produced nothing trustworthy. 30 auto-filed crash issues, 0 with a usable stack frame.

Every build now retains its symbols as a CI artifact. This is additive — Sentry stays the primary automatic path; the artifact is the manual path.

PlatformArtifactContainsRetention
iOSios-dsyms-<profile>Every .dSYM from the archive (app and frameworks — Hermes, React), plus dsym-uuids.txt and README.txt30 days
Androidandroid-symbols-<profile>native/ unstripped .so libraries, native-build-ids.txt, mapping/ R8 mapping files when minification is enabled30 days

iOS — from a .ips to a named frame ​

  1. Find the build. The crash report's Binary Images section lists each loaded binary with its UUID and load address. Match the app's UUID against builds in the Actions tab — or just find the mobile-build-native.yml run whose version/build number matches the report header.

  2. Download ios-dsyms-<profile> from that run's artifacts (Actions → the run → Artifacts). No Sentry access needed; repo read access is enough.

  3. Confirm the UUID. dsym-uuids.txt in the artifact maps every UUID to its .dSYM bundle. If the report's UUID is not in that file, you have the wrong build — go back to step 1 rather than symbolicating against the wrong symbols.

  4. Symbolicate the frame:

    bash
    atos -o <Name>.dSYM/Contents/Resources/DWARF/<Name> \
         -arch arm64 \
         -l <load address from Binary Images> \
         <frame address>

    Frames inside hermes/React come from those frameworks' own dSYM bundles, not the app's — pick the bundle whose UUID matches that frame's binary in Binary Images.

CI proves the artifact matches the shipped binary. The Verify dSYM UUIDs match the shipped IPA and stage for retention (#11000) step unzips the exported IPA, reads its main executable's LC_UUIDs with dwarfdump -u, and asserts every one is present among the archived dSYMs — failing the job (before the TestFlight submit) if any is missing. An artifact that does not match the shipped binary is worse than no artifact, because it looks like coverage.

Android — from a native crash to a named frame ​

Android release builds are not minified (no expo-build-properties plugin overrides Expo's android.enableProguardInReleaseBuilds=false default), so Java/Kotlin frames are already readable and mapping/ is normally empty. That is expected, not a fault. The files that matter are the unstripped native libraries in native/ — the Android analogue of dSYMs, and the ones that carry Hermes.

  1. Download android-symbols-<profile> from the matching workflow run.
  2. Match the crash frame's library by GNU build ID using native-build-ids.txt.
  3. llvm-addr2line -f -C -e native/<abi>/<lib>.so <offset> (or ndk-stack if you have the full logcat trace).

The Android side is deliberately advisory, not a gate. Its collection step never fails the build: the merged_native_libs layout shifts between AGP versions and there is no #10831-equivalent Android gate for it to sit behind, so a hard assertion on that path would risk bricking Android CD to protect symbols the team does not currently have at all. If it finds nothing it emits a ::warning:: and a job-summary line rather than silently publishing an empty artifact. The build-ID cross-check against the shipped APK is a warning for the same reason. Only iOS carries the blocking UUID gate.

Verifying the gate is working ​

  • A successful run's job summary shows three green checkmarks: dSYMs uploaded, dSYM verification passed (with a bundle count), and the symbolication smoke-check passed (with a UUID count).
  • To confirm end-to-end, force a native crash on a TestFlight build built after this gate shipped and check that the resulting Sentry issue resolves a full native stack trace (function names, file/line), not raw addresses.
  • mobile/ppp CI/CD reference — full mobile-build-native.yml workflow reference
  • #10589 — original dSYM upload (soft)
  • #10816 — local dSYM validity check + Sentry release health (soft)
  • #10831 — this gate (hard)
  • #10828 — parent epic: making the JSI-crash class un-shippable
  • #11000 — dSYM/symbol retention as a CI artifact (the manual, token-free route above)
  • #10964 — the live Hermes SIGSEGV that #11000 unblocks triage for

Internal documentation — org members only