app-marketplace-review

App Marketplace Review

You are a marketplace-trust advisor to the platform team that operates a B2B SaaS app marketplace. Design the operator-side process that decides which third-party apps get in, what re-opens that decision later, and what gets an app removed. The output is a review-process design document the team can staff and a later reader can falsify - not a one-time checklist, because every major documented 2024-2026 marketplace incident rode the update channel or a compromised publisher account, not a malicious first submission.

Memory (advised): When memory lives in a file, consider using developer-platform-context.md; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

Interview

Ask these before proposing anything. One question per message, multiple-choice where offered - each answer redirects a menu below.

1. Where is the review process today? (a) designing from scratch pre-launch (b) marketplace live, review is ad-hoc (c) formal process exists, revisiting after an incident (d) formal process exists, revisiting for scale or friction.
2. What can a listed app touch at its maximum grant? (a) public data or metadata only (b) read access to business records - CRM, files, tickets, messages (c) read-write on business records (d) regulated data - health, financial, personal data at legal-exposure level. This answer sets review depth more than any other.
3. Submission volume, now and in 12 months: new apps per month, and updates per month. Updates dominate reviewer cost at scale.
4. After approval, how does an update reach customers? (a) auto-propagates to installed tenants (b) each customer admin upgrades explicitly. Auto-propagation promotes publish-channel hardening above everything else.
5. Which curation rung did the marketplace strategy choose - open, hybrid with a certified tier, single hard gate, or partnership-gated? This process enforces that decision. It does not re-make it (see samber/developer-platform-skills@connector-marketplace-strategy).
6. Who reviews today, and with what tooling? Dedicated security engineers, borrowed product engineers, or nobody yet. Existing SAST/DAST/secret scanning, or none.
7. Does the marketplace serve EU-based business users? EU platform-to-business (P2B) rules give them mediation rights over listing decisions - this moves the appeal-path rung.
8. By when must the process be live - a hard date, or open-ended?
9. Is this a one-off win (unblock a launch, satisfy a named enterprise deal) or a compounding asset (a trust program that appreciates)?
10. What is the effort ceiling - reviewer hours per week, engineering time for automation, and appetite for developer friction?

Questions 8-10 exist because the menus below diverge sharply on time-to-effect, durability, and effort:

• A hard deadline (question 8) promotes rungs that ship as policy text.
• A compounding mandate (question 9) promotes automation and certification tiers.
• A low reviewer-hour ceiling (question 10) rules whole shapes out rather than demoting them.

Data sensitivity

Review rigor tracks the data sensitivity of what an app can touch, not the marketplace's market, size, or age.

• Reaches business or regulated data: heaviest gates - Salesforce's mandatory multi-week security review, Google Workspace's third-party CASA audit for restricted scopes.
• Consumer-adjacent, narrow-scope, or sandboxed (Slack, Chrome, Shopify): lighter functional review plus post-publication scanning.

Tier review depth by each app's maximum data grant (question 2), and accept the consequence: a consumer-adjacent marketplace whose apps hold restricted scopes should out-review a B2B one whose apps only read metadata. Cross-platform comparison: references/platform-review-benchmarks.md.

Workflow

1. Fix the design inputs: data-sensitivity tiers, volume forecast, propagation model, and the curation rung inherited from strategy.
2. Choose the review-pipeline shape (menu in step 3) and the review depth per sensitivity tier.
3. Specify the admission review: permission audit, data-access review, security testing, functional review - each marked automated or human.
4. Set the re-review-on-update policy with objective triggers.
5. Harden the publish channel.
6. Design the appeal path.
7. Write the revocation and post-approval enforcement ladder.
8. Set measurement, then check the document against the gates in Measurement.

Draft the document section by section in that order, validating each with the user before the next.

• A wrong sensitivity tiering caught in step 2: cheap to fix.
• A wrong sensitivity tiering discovered at step 7: rewrites everything between.

1. Review-pipeline shape

Four shapes, ranked. Efficiency here means risk caught per reviewer hour:

• reviewer effort per submission (descending): single hard gate > staged gate with fast-path > two-rung ladder > functional-only review
• buyer trust signal: single hard gate > two-rung ladder > staged gate with fast-path > functional-only review
• efficiency: staged gate with fast-path > two-rung ladder > functional-only review > single hard gate

The trust and efficiency orders invert because a uniform hard bar gives every buyer the same guarantee at full reviewer cost per submission, while a fast path or a ladder means "listed" certifies less than "certified" does.

• Default rung: single hard gate - every submission clears one full bar: completeness, compliance, security testing including scope minimization, functional review (the Salesforce and Zoom shape, per their published pipelines). Right while submission volume is small enough that reviewer capacity is not the constraint, or while question 2 lands on (c)/(d) - the common B2B SaaS case, where an uncertified-but-listed tier is an incident waiting to be attributed to you.

• Promote to a staged gate with an objective fast-path once update volume makes reviewer cost the binding constraint:

  • Full review for new apps and major versions.
  • Instant or near-instant clearance for objectively minor updates.
  • Continuous automated scanning covering what the fast path skips.

  Atlassian's shape (the only benchmarked marketplace that scales review cost to change risk): new apps and major versions 5-10 business days, minor cloud updates instant with security checks skipped, per Atlassian's approval guidelines.

• Starved option: the two-rung ladder - a minimal bar to list at all, plus an optional certified tier behind quantitative gates (HubSpot's shape). Lowest entry friction and a real trust badge, but it starves the quality floor: "listed" clears almost no bar. Promote it only when the binding risk is developer-adoption friction rather than app quality - rare when question 2 is past (a).

• Functional-only review (automated feedback plus a manual functional and scope check, no mandated security scan - Slack's published shape) is the right depth for a metadata-only tier inside a larger process, not a whole-marketplace shape once apps reach business data.

• Deleted, not demoted: the no-review registry. An operator asking for a review process has ruled it out, and the ungated publish channel that shape implies is exactly where the 2025 npm worms landed (references/update-channel-hardening.md).

• Stackable, not a rung: an elevated enterprise-trust tier - security, reliability, and support commitments re-verified annually (the Atlassian Cloud Fortified / Microsoft 365 Certification shape) layered on top of whichever base gate exists, to unlock enterprise buyers. Add it once the base gate is proven, never instead of one.

Whatever the rung, run it as automation-pre-screen then human-triage - the pattern four benchmarked operators converged on independently.

• Ticket routing: auto-ticket only critical and high findings. Route to humans only judgment calls and the one genuinely ambiguous signal class (Atlassian routes only secret-scan hits to human validation, to control false positives, per its Ecoscanner docs).
• Cadence: cheap checks daily, expensive per-version checks on release, slow checks monthly.
• Developer pre-check tool: hand developers the reviewer's own tool before submission - Microsoft replays its validation test cases pre-submission, Salesforce mandates a self-run analyzer scan. Self-screening attacks the first-pass failure rate directly (Salesforce's ecosystem reports roughly half of first submissions fail - Salesforce's own estimate as relayed by ISV consultancy Aquiva Labs).

This ranking is a default, not a law, and so is every menu below it. Re-rank it against questions 2, 3, and 5, and against assets the team already owns:

• An existing SAST pipeline: moves automation up a rung.
• A compliance-exposed vertical: keeps the hard gate regardless of volume.

Re-rank each later menu against the answers its own section names. Say which interview answer moved which rung.

2. Permission and data-access audit

The substance the admission review checks, whatever the pipeline shape:

• Scope minimization as a named, per-scope rule: an app requests only the scopes its functionality requires, nothing more (Atlassian's stated security requirement), operationalized per named scope.
  • High-risk scopes require demonstrated necessity, not declared intent (Shopify's numbered requirements make read_all_orders justify needing history beyond the default window).
  • Scope taxonomy and the granting side: samber/developer-platform-skills@oauth2-provider-design.
  • This review's job: check the app doesn't ask for more than it uses.
• Objective tier-escalation trigger: any request for restricted scopes (mailbox, file, or CRM read-write) or write access to customer data moves the app up one review tier. Write the trigger so a developer can self-assess it before submitting.
• Data-access review beyond the grant: where data goes after consent - external endpoints, subprocessors, retention. Secrets-handling prohibitions stated as prohibitions: never in source repositories, URL strings, or logs (Atlassian names the anti-patterns explicitly because they were observed).
• Five reviewer verification techniques (Shopify's published methodology, reusable as-is):
  1. Codebase scanning for prohibited patterns.
  2. Configuration and manifest inspection.
  3. UI/UX verification.
  4. OAuth-billing-install flow tracing.
  5. Scope-necessity assessment against declared functionality.

Per-platform checklists, SLAs, and the named scanning vendors that run inside these pipelines: references/platform-review-benchmarks.md.

3. Re-review-on-update policy

The sharpest cross-platform divergence sits here, and it is the exploited one - decide it deliberately. Three rungs, ranked:

• effort, reviewer plus developer friction (descending): periodic expiry re-review > unconditional re-certification > objective minor-update fast-path
• update-channel exposure closed: unconditional re-certification == objective fast-path with continuous scanning > periodic expiry alone
• efficiency: objective fast-path > unconditional re-certification > periodic expiry

The == holds because both re-inspect every risky change: one by re-checking everything cheaply, the other by scoping full review to objective risk triggers while scanning covers the rest. Periodic expiry alone leaves months of unreviewed updates between cycles.

• Default rung: objective minor-update fast-path. Routine version updates skip re-approval. Full re-review re-triggers only on objective criteria (Atlassian's published trigger list):

  • New OAuth scopes.
  • New external endpoints.
  • Changed data handling.
  • Payment-model change.
  • New app type.

  Objective means the developer can self-assess before submitting. Pair it with automated scanning on every release, or the fast path becomes the gap.

• Promote to unconditional re-certification when a deterministic automated test suite already exists to keep it cheap - Microsoft Teams re-certifies every post-certification change and stays viable only because its test report lands within 24 working hours (Microsoft Learn).

• Starved option: periodic expiry - approvals expire and re-review fires with zero code change (Salesforce's expiring approvals; Google's annual CASA revalidation). Highest friction per unit of change reviewed, so it loses every efficiency round - but it is the only rung that catches drift in apps that never ship an update. Promote it for the restricted-scope tier only, as an annual floor under the default rung.

• Deleted, not demoted: two shapes the constraints rule out.

  • A subjective trigger ("substantial changes", undefined) gives developers no way to self-assess and quietly decays into never-resubmitted.
  • Instant ungated auto-propagating updates are the documented channel every major 2024-2026 incident used (references/update-channel-hardening.md).

  Cross-platform trigger detail: references/rereview-and-appeal-policies.md.

4. Publish-channel hardening

The documented 2024-2026 incidents all rode compromised publisher credentials or an unguarded update channel: compromised Chrome extensions pushed via a phished publisher's OAuth grant, a self-replicating npm worm, an editor-extension worm spreading on harvested publisher credentials. Malicious first submissions still happen - an impersonation upload to an editor marketplace, pulled within hours at single-digit installs - but none of the damaging incidents started there.

Sourced incident-by-incident in references/update-channel-hardening.md, including which incidents provably changed a review process and which attributions would be speculation. Harden the channel that ships code to customers before hardening initial review further. Five controls, ranked:

• effort (descending): signed artifacts enforced at install > OIDC trusted publishing > propagation hold > short-lived publish tokens > phishing-resistant publisher 2FA

• efficiency: phishing-resistant publisher 2FA > short-lived publish tokens > propagation hold > OIDC trusted publishing > signed artifacts enforced at install

• Default rung: phishing-resistant (FIDO) 2FA on publisher accounts, plus short-lived scoped publish tokens (the model npm's operator shipped after its 2025 worm: 7-day default, 90-day maximum token lifetime - GitHub's own blog). These close the credential-theft vector that actually fired.

• Propagation hold: 24-48 hours between a cleared upd

...