← Back to Site
Hidden Insight Labs·Insight Report
Sample Document Small Evaluation 2-Pass Testing
Insight Report: Core Loop & Monetisation UX
[CLIENT APP]  ·  Mobile Application  ·  Tester panel: 21 participants across 2 passes  ·  Flows evaluated: Install → Tutorial → First session → In-app purchase prompt  ·  Devices: iOS and Android, 14 device configurations  ·  Testing window: ~8 days
Report Date
April 2026
Engagement Type
Small Evaluation
Total Testers
21 (2 passes)
Pass 1 Panel
14 testers
Pass 2 Panel
7 testers
Device Configs
14 configurations
Integration Required
None
Pass 1 Completion
8 / 14 (57%)
Pass 2 Completion
6 / 7 (86%)
Friction Points (P1)
16
Friction Points (P2)
6 remaining
Issues Resolved
10 of 16
01

Severity Summary — Pass 1 vs Pass 2

Pass 1 — Initial Testing (14 testers)
Pass 2 — After Fixes (7 testers)
Overall assessment: Pass 1 identified three critical blockers concentrated in the permissions flow and monetisation prompt. Following the client's patch, Pass 2 confirmed all three critical issues resolved — completion rate improved from 57% to 86%. Six issues remain open, including two high-severity findings in the tutorial gesture system and the map icon legibility problem. These are documented with full recommendations in Section 08. The two-pass structure provided clear before/after evidence of fix effectiveness, and enabled the team to reprioritise development effort based on verified user impact.
02

Tester Panel & Device Coverage

Pass 1 recruited 14 testers across casual, mid-core, and core gaming profiles. Pass 2 used a 7-tester subset focused on verification of high-priority fixes, weighted toward the casual profile that experienced the highest abandonment in Pass 1.

iPhone 14
iOS 17.4
Casual · Pass 1 & 2
iPhone 13 mini
iOS 16.7
Casual · Pass 1 & 2
iPhone 12
iOS 15.8
Mid-core · Pass 1
Samsung Galaxy A54
Android 14
Core · Pass 1 & 2
Samsung Galaxy A23
Android 13
Casual · Pass 1
Pixel 7
Android 14
Mid-core · Pass 1 & 2
Pixel 6a
Android 13
Mid-core · Pass 1
OnePlus 11
Android 13
Core · Pass 1
Motorola G82
Android 12
Casual · Pass 1
Xiaomi Redmi Note 12
MIUI 14 / Android 13
Mid-core · Pass 1
OPPO A78
ColorOS / Android 13
Casual · Pass 1
iPad Air (5th gen)
iPadOS 17.3
Mid-core · Pass 1
Samsung Galaxy Tab A8
Android 13
Casual · Pass 1
iPhone SE (3rd gen)
iOS 17.2
Casual · Pass 2
Coverage note: The Xiaomi and OPPO devices surfaced a UI scaling issue (F-14) not visible on any other tested configuration — a finding that would not have appeared in a smaller pilot panel. MIUI's aggressive background app management also caused one unexpected session termination, logged as F-15.
03

Testing Iterations

Pass 1
Initial Evaluation
Testers14
Duration~5 days
Completion57%
Issues found16
Critical3
Client patch window
Client team addressed
10 of 16 findings
Pass 2
Fix Verification
Testers7
Duration~3 days
Completion86%
Issues remaining6
Critical0
04

Flow Friction Map — Pass 2 State

Critical — caused abandonment
High / medium friction remaining
Resolved — confirmed in Pass 2
No significant friction
Step 1
App Store / Install
Step 3
Character Select
Step 6
First Dungeon
05

Friction Point Log — All Issues

Resolved rows are dimmed. Status reflects Pass 2 verification. Frequency shown is Pass 1 (n=14) unless marked P2.

# Location Description Severity (P1) Frequency (P1) Status
F-01 Launch & Permissions Camera, microphone, and contacts requested on first launch before any gameplay context. Caused 5 of 14 testers to uninstall immediately. Critical
5/14
Resolved
F-02 IAP Prompt Monetisation overlay appeared after first dungeon death, before the player reached any reward. 9 of 14 testers reacted negatively; 3 uninstalled at this point. Critical
9/14
Resolved
F-03 IAP Prompt No dismiss option visible on IAP overlay for 4 seconds. Testers assumed the app had locked them into a purchase flow. Critical
14/14
Resolved
F-04 Combat Tutorial Two-finger diagonal swipe mechanic taught by static icon only — no animation, haptic feedback, or retry prompt. All completing testers attempted this incorrectly multiple times. High
9/14
Open
F-05 Combat Tutorial Tutorial countdown timer runs during instructional overlays, creating pressure to skip reading. 6 testers did not read combat instructions before proceeding. High
8/14
Open
F-06 Map Navigation World map dungeon icons unlabeled. Testers could not distinguish story from challenge dungeons; 7 entered challenge content with starter equipment. High
7/14
Partial
F-07 Map Navigation "Return to base" button not visible on small displays (≤5.4"). Hardware back gesture caused accidental app close on iOS. High
4/14
Resolved
F-08 IAP Prompt Currency naming ("Vault Gems" vs "Vault Coins") confused testers — unclear which was premium vs earned. 5 testers were uncertain whether they were spending real money. High
5/14
Resolved
F-09 Combat Tutorial Enemy attack audio warnings too quiet at device default volume. Visual fallback absent. All testers missed audio cues in ambient conditions. Medium
14/14
Open
F-10 First Dungeon Death screen shows no path back to dungeon selection — only "Try Again" or "Home." 4 testers assumed they'd lost progress and quit the session. Medium
4/14
Resolved
F-11 Character Select Character stat tooltips require undiscoverable long-press. All testers chose characters by appearance; none read stats before committing. Medium
14/14
Open
F-12 First Dungeon Reward animation plays before loot is actually credited to inventory. 3 testers navigated to inventory mid-animation and saw empty slots, assumed a bug. Medium
3/14
Resolved
F-13 Map Navigation No visual indicator of player level or progression status on map screen. Testers returning to map after dungeon completion couldn't assess what to do next. Medium
8/14
Open
F-14 Combat Tutorial UI elements overflow their containers on MIUI / ColorOS. Ability icons partially obscured on Xiaomi and OPPO devices. Not reproducible on stock Android or iOS. Low
2/14
Open
F-15 First Dungeon MIUI background app management terminated session during dungeon load. No session restore on relaunch — tester returned to home screen with no progress saved. Low
1/14
Resolved
F-16 Character Select Confirmation dialog for character selection uses "OK" as the confirm label — not the character's name. Low urgency; caused brief hesitation in 4 testers. Low
4/14
Resolved
06

Cross-Tester Patterns

With 14 testers in Pass 1, several behavioural patterns emerged across player profiles that would not be visible at pilot scale. These patterns inform the prioritisation in Section 08.

Casual players — distinct failure mode

Permissions = Trust Signal

All 5 casual-profile testers who uninstalled cited the permissions prompt as the trigger. Core and mid-core testers showed no abandonment at this step — they were accustomed to the pattern. This profile-specific finding would not be statistically visible with fewer than 8–10 testers and demonstrates why panel size matters for risk surface coverage.

All profiles — universal friction point

Gesture Mechanics Require Animation

Across all player profiles, the two-finger swipe failure pattern was consistent — though core gamers recovered faster. The shared failure suggests this is a UX design gap, not a familiarity gap. The static icon instruction format fails regardless of gaming experience level.

Mid-core players — navigation confusion

Map Legibility Scales with Engagement

Mid-core testers were the most likely to spend time on the map trying to understand the icon system before selecting — and the most likely to feel frustrated when they couldn't decode it. Casual testers simply tapped the nearest icon. Core testers explored methodically. The mid-core pattern is the highest-risk for early churn from this group.

All profiles — IAP response

Premature Monetisation Damages Trust Universally

The IAP overlay triggered negative verbal reactions from 9 of 14 testers across all profiles. The 4-second dismiss delay compounded this — testers assumed dark pattern intent. After the fix in Pass 2 (delay removed, prompt timing adjusted), zero testers cited the IAP flow as a negative.

07

Completion Analysis

Pass 1 — Completion by Player Profile
Core gamer
4/4
Mid-core
3/4
Casual
1/6
Overall
8/14
Pass 2 — Completion by Player Profile
Core gamer
2/2
Mid-core
2/2
Casual
2/3
Overall
6/7
Note on casual completion rate: The one non-completion in Pass 2 (casual profile, iPhone SE) was attributable to F-04 — the gesture mechanic still unresolved. This tester abandoned the tutorial after 5 failed swipe attempts. The remaining casual testers in Pass 2 completed the flow, confirming that the permissions fix (F-01) was the primary driver of Pass 1 casual abandonment.
08

Prioritised Recommendations — Open Issues

1
Replace static gesture icon with animated instruction + haptic feedback loop
Open

The power slash two-finger diagonal swipe should be demonstrated through a looping animation at native resolution, showing finger placement and direction. Add a subtle haptic confirmation on correct execution, and a "try again" coaching prompt after 2 failed attempts. This is the highest-remaining-priority fix — it affected 9 of 14 Pass 1 testers and remains the primary driver of tutorial abandonment in the casual segment. Estimated dev effort: 1–3 days (animation asset + haptic trigger).

Affects: F-04 Profile Impact: All Effort: Medium Impact: High
2
Pause countdown timer when instructional overlays are displayed
Open

The tutorial timer should pause automatically whenever any instructional modal or overlay is shown and resume only when the tester dismisses it or begins interacting. This is a behavioural design fix, not a visual one — the timer can remain visible but inactive. Consider also offering a "take your time" mode as a first-launch option for players who indicate they are new to action RPGs.

Affects: F-05 Profile Impact: Casual, Mid-core Effort: Low Impact: High
3
Add text labels and level-appropriate difficulty indicators to all dungeon map icons
Partial fix

The client's patch added colour coding to dungeon icons, which was insufficient — 3 of 7 Pass 2 testers still entered incorrect content. Icons must include a short text label visible on the map view (not only on tap), plus a level range indicator (e.g. "Lv 1–5"). A confirmation prompt for out-of-range dungeons ("This dungeon is recommended for Lv 10+. You are Lv 2. Continue?") should be implemented as a short-term safeguard while the icon system is fully redesigned.

Affects: F-06, F-13 Profile Impact: Mid-core Effort: Low–Medium Impact: High
4
Add visual fallback for all audio-based warnings; increase default mixing level
Open

Enemy attack warnings require a visual component — a brief screen edge flash or ability icon pulse — that functions without audio. Increase default mixing for warning sounds by approximately 6dB relative to background music. This is an accessibility improvement as much as a usability fix and should be treated as baseline quality for a mobile release.

Affects: F-09 Profile Impact: All Effort: Low Impact: Medium
5
Surface character stats without requiring long-press; add "i" info affordance
Open

Add a visible info icon (ⓘ) to each character card that opens the stat tooltip. The long-press interaction can remain but should not be the only path to this information. Consider displaying one or two key stats (e.g. Attack, Defence) directly on the card without requiring any interaction at all, to make the character choice feel more informed from the outset.

Affects: F-11 Profile Impact: All Effort: Low Impact: Medium
6
Fix UI overflow on MIUI and ColorOS; test against manufacturer skin constraints
Open

The ability icon overflow on Xiaomi and OPPO devices is caused by non-standard safe area insets in MIUI 14 and ColorOS 13. These manufacturer skins represent a significant share of the Android install base in key markets (South/Southeast Asia, parts of Europe). Recommend adding MIUI and ColorOS to the standard device test matrix and auditing all UI containers against their safe area specifications.

Affects: F-14 Profile Impact: Device-specific Effort: Medium Impact: Low–Medium
09

Pass 2 Fix Verification Summary

The following table summarises verification results for each issue the client addressed between passes. Pass 2 testers were not informed which issues had been fixed.

F-01 — Permissions timing
✓ Fully resolved — 0 uninstalls at launch
F-02 — IAP prompt timing
✓ Fully resolved — 0 negative reactions
F-03 — IAP dismiss delay
✓ Fully resolved — dismiss visible on load
F-06 — Map icon legibility (partial fix)
⚠ Partially resolved — colour added but insufficient
F-07 — Return to base button
✓ Fully resolved — button visible on all tested devices
F-08 — Currency naming clarity
✓ Fully resolved — no confusion in P2
F-10 — Death screen navigation
✓ Fully resolved — "Select Dungeon" option added
F-12 — Reward animation timing
✓ Fully resolved — inventory updates before animation
F-15 — MIUI session termination
✓ Resolved — session restored on relaunch
F-16 — Confirm dialog label
✓ Resolved — button now shows character name