QA Plan¶
Quality strategy for the Yam3at MVP across the Laravel API, Next.js web app, and Flutter mobile apps. QA is embedded from Sprint 1 — acceptance criteria live in the SRS modules and the backlog; this plan defines how they are verified.
Test pyramid¶
flowchart TB
E["Exploratory & release testing<br/>manual, bilingual, on real devices"]
P["E2E: Playwright (web) + Flutter integration tests (mobile)<br/>~60–80 scenarios, staging nightly"]
A["API/feature tests: Pest (Laravel HTTP tests)<br/>every endpoint, auth matrix, state machines"]
U["Unit tests: Pest/PHPUnit + Dart test<br/>money math, validators, policies, services"]
E --- P --- A --- U| Layer | Tooling | Owner | Gate |
|---|---|---|---|
| Unit | Pest/PHPUnit (PHP), package:test (Dart), Vitest (web utils) |
Developers | Required in CI per PR; money/state-machine logic 100% branch-covered |
| API / feature | Pest HTTP tests against SQLite/MySQL test DB, PSP + FCM + SMS faked | Backend devs | Required in CI per PR |
| Contract | OpenAPI schema validation of responses in Pest; generated client types for web | Backend + web | CI per PR |
| E2E web | Playwright (Chromium + WebKit), ar and en projects |
QA | Nightly on staging; smoke subset on every deploy |
| E2E mobile | Flutter integration_test on Firebase Test Lab (Android matrix + iOS) |
QA + mobile dev | Nightly; pre-release full run |
| Exploratory | Session-based test charters, bilingual testers | QA | Per sprint + pre-release |
Rules: no test may call the real PSP, SMS gateway, or FCM outside the dedicated sandbox suites. Every bug fixed gets a regression test at the lowest layer that can catch it.
Environments¶
| Env | Purpose | Data | Payments | Notes |
|---|---|---|---|---|
local |
Development | Seeders + factories | PSP mock (in-process fake) | Docker Compose, Mailpit for email |
ci |
Per-PR pipeline | Ephemeral, migrated fresh | PSP mock | GitHub Actions |
staging |
Integration, E2E, UAT | Synthetic seed set (see below), reset weekly | MyFatoorah/Tap sandbox (KNET test cards) | FCM real w/ test app IDs; SMS to whitelisted numbers only |
production |
Live | Real | Live KNET/cards | Feature flags for dark launches; smoke tests post-deploy |
Test-data strategy¶
- Factories/seeders produce a deterministic staging world: 40 vendors across 8 categories and 4 plan tiers (incl. one at each quotation limit boundary), 20 customers, events at every lifecycle stage, RFQs in all 8 states, bookings in all 5 states, reviews pending moderation.
- Bilingual by default: every seeded entity has Arabic and English content, including long Arabic strings, mixed-direction text (
قاعة ABC هال), and emoji, to exercise RTL/truncation. - Money fixtures intentionally use awkward KWD values:
0.750,12.345,999.999, and per-line totals that only reconcile with half-up 3dp rounding. - OTP bypass: staging accepts a fixed OTP (
000000) for whitelisted test numbers; never enabled in production (guarded by env + config test). - PSP sandbox cards: documented set for success, decline, 3-DS challenge, and timeout; webhook replay tool in the admin dev menu on staging.
- No production data is ever copied to lower environments. Support reproductions use synthetic look-alikes.
Highest-value test cases¶
Phase: S = smoke (every deploy), R = regression (nightly), P = pre-release full pass.
| ID | Area | Scenario | Expected | Phase |
|---|---|---|---|---|
| TC-001 | Auth/OTP | Register with valid +965 number, correct OTP | Account created, session issued, profile step shown | S |
| TC-002 | Auth/OTP | Wrong OTP 5 times | Code invalidated, lockout message, no session | R |
| TC-003 | Auth/OTP | Request 4th OTP within 15 min | 429 with retry-after; only 3 SMS sent | R |
| TC-004 | Auth/OTP | Expired OTP (>5 min) submitted | Rejected with localized error | R |
| TC-005 | Auth | Google and Apple sign-in, new + returning user | Account linked once; no duplicate accounts by email | P |
| TC-006 | Auth | Access vendor-only endpoint as customer | 403; audit not required, no data leak in body | S |
| TC-007 | Vendor approval | Submit application with license PDF | Status Pending review; admin queue shows item with license preview |
S |
| TC-008 | Vendor approval | Admin approves application | Vendor notified (push+email); profile editable; audit log has admin ID + timestamp | S |
| TC-009 | Vendor approval | Admin rejects without reason | Blocked by validation; reason mandatory | R |
| TC-010 | Vendor approval | Vendor resubmits after "request changes" | Returns to queue flagged as resubmission with diff of changed fields | R |
| TC-011 | Vendor approval | Unapproved vendor tries to publish profile | Blocked with explanatory state; not in search index | R |
| TC-012 | Subscription limits | Basic vendor sends quotes up to monthly limit | Quote N allowed; quote N+1 shows upgrade prompt, API returns 422 entitlement error | S |
| TC-013 | Subscription limits | Limit counter resets at billing-cycle boundary (not calendar month) | New cycle starts count at 0 | R |
| TC-014 | Subscription limits | Upgrade mid-cycle after hitting limit | New limit applies immediately; pro-rated charge correct to 3dp | R |
| TC-015 | Subscription limits | Downgrade with usage above new tier's limit | Downgrade scheduled for next cycle; current cycle unaffected | R |
| TC-016 | RFQ state machine | Customer sends RFQ to 3 vendors | 3 vendor threads Pending; vendors notified; event workspace shows RFQ |
S |
| TC-017 | RFQ state machine | Vendor quotes → customer accepts | Accepted thread Accepted; siblings auto-Rejected; draft booking created |
S |
| TC-018 | RFQ state machine | Vendor tries to quote an Expired RFQ thread |
409 state-machine error; no quote stored | R |
| TC-019 | RFQ state machine | Quote passes validity date | Auto-Expired by scheduler; both parties notified; cannot be accepted |
R |
| TC-020 | RFQ state machine | Customer cancels RFQ while one thread is Negotiating |
All non-terminal threads → Cancelled; vendors notified |
R |
| TC-021 | RFQ state machine | Vendor at plan limit receives RFQ | Can read RFQ; cannot submit quote (entitlement gate, TC-012) | R |
| TC-022 | Payments/webhooks | Successful KNET deposit checkout (sandbox) | Booking Pending→Confirmed; receipt issued; customer + vendor notified |
S |
| TC-023 | Payments/webhooks | Same payment webhook delivered twice | Second delivery is a no-op (idempotency key); single confirmation, single receipt | S |
| TC-024 | Payments/webhooks | Webhook with invalid signature | 401/rejected; alert raised; no state change | R |
| TC-025 | Payments/webhooks | Payment fails / user abandons checkout | Booking stays Pending with retry CTA; no orphan transactions |
R |
| TC-026 | Payments/webhooks | Webhook arrives before redirect return (race) | Final state correct regardless of order; no double processing | P |
| TC-027 | Payments/webhooks | Admin partial refund via PSP | Refund recorded to the fils; booking → per policy; audit logged | R |
| TC-028 | KWD rounding | Quote with line items summing to 12.3455 | Half-up rounding at 3dp applied once at total; UI, DB, PSP amount identical | S |
| TC-029 | KWD rounding | Display of KWD 0.750 across web/mobile/AR locale | Always 3 decimals (0.750, ٠٫٧٥٠ per locale rules); never 0.75 |
R |
| TC-030 | KWD rounding | Subscription VAT-free total + promo (Scale) math | Stored integer fils == displayed amount == PSP charge | P |
| TC-031 | RTL | Full RFQ wizard in Arabic on web (Playwright ar project) |
Layout mirrored; steppers, chevrons, progress flow right→left; no clipped text | S |
| TC-032 | RTL | Mixed AR/EN vendor name in chat bubbles and notifications | Bidi renders correctly (no scrambled brackets/numbers) | R |
| TC-033 | RTL | Flutter app language switch AR↔EN at runtime | All screens re-render mirrored without restart artifacts | R |
| TC-034 | RTL | Quote comparison table in Arabic with 3dp prices | Columns mirrored, numerals consistent, sort still correct | P |
| TC-035 | Notifications | New quote received (push + email + in-app) | All three delivered; push deep-links to quote; in-app marks read | S |
| TC-036 | Notifications | User disables email channel in preferences | Email suppressed; push and in-app unaffected | R |
| TC-037 | Notifications | Notification templates in user's language | Vendor (AR) and customer (EN) each get their own language for the same event | R |
| TC-038 | Reviews | Review attempt without a Completed booking |
Blocked; UI explains verified-review rule | R |
| TC-039 | Booking lifecycle | Auto-complete N days after event date | Booking → Completed; review prompt sent |
R |
| TC-040 | Messaging | Send 10 MB+1 file and .exe file in chat | Both rejected with clear errors; 10 MB PDF succeeds | R |
| TC-041 | Search | Arabic typo search ("تصوير" misspelled) returns photographers | Meilisearch typo tolerance returns expected vendors; featured ordering respected | R |
| TC-042 | Security | IDOR probe: customer A requests customer B's event/quote/booking by ID | 403/404 on all; verified across all resource types | P |
Non-functional testing¶
Load (k6 against staging, production-like sizing)
| Target | Threshold |
|---|---|
| API p95 latency (browse/search/RFQ endpoints) | < 300 ms at 200 concurrent users |
| Search (Meilisearch-backed) p95 | < 150 ms |
| Checkout/webhook processing p95 | < 500 ms; zero lost webhooks at 20/s burst |
| Sustained load | 200 concurrent users, 30 min, error rate < 0.1% |
| Launch spike | 5× baseline for 10 min with graceful degradation (queues drain < 5 min after) |
| Push fan-out | 10k notifications queued and delivered < 2 min via Horizon |
Security
- OWASP ZAP baseline scan in CI weekly; full active scan pre-release on staging.
- Dependency audit (
composer audit,npm audit, Dependabot) — Critical/High block release. - Static analysis: Larastan level 8+, ESLint security rules, Dart analyzer.
- Manual checks pre-launch: authz matrix per role (TC-042 expanded), OTP/rate-limit abuse, file-upload polyglot tests, webhook signature enforcement, secrets scan.
- PII review: exports/logs never contain phone, license documents, or payment tokens.
Other NFR passes: accessibility smoke (focus order in RTL, contrast, TalkBack/VoiceOver on core flows), Lighthouse ≥ 90 performance/SEO on public category×city pages, offline/poor-network behavior on mobile (airplane-mode mid-checkout must not corrupt state).
Release checklist¶
- All S + R suites green on the release candidate; P pass executed and signed off
- Zero open Critical/High bugs; Mediums triaged with owner + date
- Migration dry-run against a production-size snapshot; rollback script tested
- PSP sandbox regression (TC-022–TC-027) plus one real production micro-transaction and refund
- Bilingual review of all new user-facing strings (native Arabic reviewer)
- RTL visual audit of changed screens (designer sign-off)
- Store builds: version bump, changelog AR/EN, phased rollout configured (10% day 1)
- Monitoring dashboards + alerts updated for new endpoints; Sentry release tagged
- Feature flags default states confirmed; kill switches documented
- Support/ops briefed on changes (Operations Handbook updated if SOPs changed)
- Go/no-go recorded with attendees and decision
Bug severity & SLA matrix¶
| Severity | Definition | Examples | Fix SLA | Release gate |
|---|---|---|---|---|
| Critical (S1) | Money wrong, data loss, security breach, platform down, nobody can pay/login | Double charge; webhook confirms wrong booking; OTP bypass; KWD rounding error on invoices | Hotfix immediately, workaround < 4 h | Blocks release & triggers incident process |
| High (S2) | Core flow broken for a user segment, no workaround | Vendors on iOS can't submit quotes; Arabic checkout button unreachable | Fix < 2 business days | Blocks release |
| Medium (S3) | Flow impaired but workaround exists; cosmetic issue on a core screen | Comparison table mis-sorts; notification arrives without deep link | Next sprint | Ship with noted exception |
| Low (S4) | Minor cosmetic, rare edge case, internal tooling | Misaligned icon in admin; typo in English FAQ | Backlog, batched | Never blocks |
Severity is set by QA, contestable by the tech lead; anything touching payments, entitlements, or auth is at minimum S2. Every S1/S2 requires a post-fix regression test and a one-paragraph root-cause note in the bug ticket.