Testing Framwork #17

Merged

an.di merged 83 commits from adilallo/enhancement/TestingFramework2 into main 2025-09-03 18:50:40 +00:00

Conversation 0 Commits 83 Files Changed 254

+359221 -56222

an.di commented 2025-08-30 02:30:33 +00:00

Owner

Copy Link

Copy Source

Comprehensive Testing Framework Overhaul and CI Infrastructure

Overview

This PR establishes a complete, production-ready testing framework for the CommunityRule application, consolidating Storybook configuration, implementing comprehensive E2E testing with Playwright, and creating a robust CI/CD pipeline for the Gitea runner environment. The changes transform the project from basic unit tests to a multi-layered testing strategy that ensures code quality, visual consistency, and performance across all browsers and devices.

Changes

��️ Testing Framework Architecture

• Multi-layered testing strategy: Unit (Vitest), Integration, E2E (Playwright), Visual Regression, Performance (Lighthouse CI), and Storybook testing
• Cross-browser compatibility: Full test coverage across Chromium, Firefox, WebKit, and Mobile projects
• Test consolidation: Removed redundant visual regression tests from E2E files, establishing clear separation of concerns

�� Storybook Consolidation & Configuration

• Environment-aware configuration: Automatic detection between local development and production/CI environments
• GitHub Pages integration: Base path configuration for /communityrulestorybook/ deployment
• Testing integration: @storybook/test-runner and @storybook/test for automated component testing
• Accessibility addons: @storybook/addon-a11y for WCAG compliance validation
• Viewport testing: Responsive design validation across multiple breakpoints

🧪 E2E Testing Infrastructure

• Playwright configuration: Optimized for CI environment with deterministic rendering
• Responsive testing: Comprehensive breakpoint coverage (xs, sm, md, lg, xl, 2xl, 3xl, 4xl, full)
• Functional test coverage: 77 tests across 4 test suites (homepage, user-journeys, header.responsive, footer.responsive)
• Visual regression separation: Dedicated visual-regression.spec.ts with 23 screenshot tests
• Accessibility testing: ARIA role validation and screen reader compatibility

🎨 Visual Regression Testing

• Deterministic rendering: Timezone (UTC), locale (en-US), font rasterization settings
• Layout stabilization: settle() helper function for consistent screenshot timing
• CSS normalization: Deterministic CSS for caret, scrollbars, and font rendering
• Browser-specific snapshots: Separate baseline images for each browser engine
• Tolerance management: Configurable pixel difference thresholds for CI environment variations

⚡ Performance Testing

• Lighthouse CI integration: Performance budgets and Core Web Vitals monitoring
• Chrome path resolution: Robust browser installation and path detection for CI
• Architecture compatibility: Fixed Node.js x64 vs Chrome arm64 mismatch on macOS runners
• Performance budgets: Configurable thresholds for TTFB, FCP, LCP, and interaction metrics

🔄 CI/CD Pipeline (Gitea Actions)

• 7 parallel jobs: Unit tests, E2E tests, Visual regression, Performance, Storybook, Lint, and Build
• Self-hosted runner: Optimized for macOS arm64 architecture
• Single-step server management: Start+test+teardown approach preventing process termination issues
• Port conflict resolution: Dedicated port 3010 for CI testing, avoiding conflicts with local development
• Gitea compatibility: Conditional GitHub vs non-GitHub configuration for caching and setup
• Baseline snapshot generation: seed-vr-snapshots job on main branch for stable visual regression baselines

🧹 Code Quality & Maintenance

• ESLint configuration: Comprehensive linting rules with Storybook integration
• Prettier formatting: Consistent code style across all test files
• TypeScript support: Full type checking for test files and configuration
• Test documentation: Comprehensive docs/TESTING.md with examples and workflows

Screenshots

How to Test

🚀 Quick Start

# Install dependencies and Playwright browsers
npm install
npx playwright install

# Run all test suites
npm test              # Unit tests with coverage
npm run e2e          # E2E tests across all browsers
npm run test:sb      # Storybook tests
npm run lhci         # Performance tests

🧪 Comprehensive Testing

# Unit testing
npm run test:watch   # Watch mode for development
npm run test:ui      # UI-based test runner

# E2E testing
npm run e2e:ui       # Playwright UI for debugging
npm run e2e:serve    # Dev server + E2E tests

# Visual regression
npm run visual:test  # Run visual regression tests
npm run visual:update # Update snapshots

# Performance monitoring
npm run performance:budget  # Budget-based testing
npm run performance:monitor # Continuous monitoring

🔍 Verification Checklist

• Unit Tests: 124 tests passing with >85% coverage
• E2E Tests: All 77 tests pass across Chromium, Firefox, WebKit, and Mobile
• Visual Regression: 23 screenshot tests generate consistent results
• Performance: Lighthouse CI scores meet budget thresholds
• Storybook: Builds successfully with GitHub Pages configuration
• Responsive Design: All breakpoints render correctly (xs through 4xl)
• Accessibility: WCAG compliance across all components
• Cross-browser: Consistent behavior in Chromium, Firefox, WebKit, and Mobile

🌐 Browser Testing Matrix

• Chromium: Full test suite validation
• Firefox: Cross-engine compatibility
• WebKit: Safari engine validation
• Mobile: Responsive design verification

Notes

🏗️ Architecture Decisions

• Testing separation: Functional E2E tests separate from visual regression for maintainability
• CI optimization: Single-step server management prevents environment drift between CI steps
• Port isolation: Port 3010 dedicated to CI to avoid conflicts with local development on 3000
• Snapshot strategy: CI-generated baseline snapshots eliminate local vs CI environment differences

🔧 CI Environment

• Self-hosted runner: Manual activation for resource management and control
• Architecture compatibility: arm64 Node.js + arm64 Chrome for optimal performance
• Gitea integration: Conditional configuration for GitHub vs Gitea environments
• Baseline management: Automated snapshot generation on main branch for stable comparisons

🚀 Future Improvements

• Test coverage expansion: Additional edge cases and accessibility scenarios
• Performance optimization: Dynamic budget adjustment based on production metrics
• Visual regression stability: Enhanced deterministic rendering for CI environments
• Storybook enhancement: Additional addons for component testing and documentation

�� Known Limitations

• Visual regression variations: Slight differences between local and CI due to font rendering
• Performance thresholds: Conservative budgets that may need tuning based on real usage
• Placeholder content: Some tests use placeholder links that don't navigate to actual pages
• CI runner management: Manual activation required for resource optimization

🔗 Dependencies & Integrations

• Playwright: E2E testing framework with visual regression capabilities
• Lighthouse CI: Performance monitoring and budget enforcement
• Storybook: Component development and testing environment
• Vitest: Fast unit test runner with React Testing Library integration
• Gitea Actions: CI/CD platform for automated testing and deployment
• Next.js 15: Application framework with built-in testing support

�� Documentation

• Comprehensive testing guide: docs/TESTING.md with examples and workflows
• Runner management scripts: Shell scripts for CI runner control
• Configuration examples: Playwright, Storybook, and CI workflow configurations
• Best practices: Testing patterns and maintenance guidelines

# Comprehensive Testing Framework Overhaul and CI Infrastructure ## Overview This PR establishes a complete, production-ready testing framework for the CommunityRule application, consolidating Storybook configuration, implementing comprehensive E2E testing with Playwright, and creating a robust CI/CD pipeline for the Gitea runner environment. The changes transform the project from basic unit tests to a multi-layered testing strategy that ensures code quality, visual consistency, and performance across all browsers and devices. ## Changes ### ��️ Testing Framework Architecture - **Multi-layered testing strategy**: Unit (Vitest), Integration, E2E (Playwright), Visual Regression, Performance (Lighthouse CI), and Storybook testing - **Cross-browser compatibility**: Full test coverage across Chromium, Firefox, WebKit, and Mobile projects - **Test consolidation**: Removed redundant visual regression tests from E2E files, establishing clear separation of concerns ### �� Storybook Consolidation & Configuration - **Environment-aware configuration**: Automatic detection between local development and production/CI environments - **GitHub Pages integration**: Base path configuration for `/communityrulestorybook/` deployment - **Testing integration**: `@storybook/test-runner` and `@storybook/test` for automated component testing - **Accessibility addons**: `@storybook/addon-a11y` for WCAG compliance validation - **Viewport testing**: Responsive design validation across multiple breakpoints ### 🧪 E2E Testing Infrastructure - **Playwright configuration**: Optimized for CI environment with deterministic rendering - **Responsive testing**: Comprehensive breakpoint coverage (xs, sm, md, lg, xl, 2xl, 3xl, 4xl, full) - **Functional test coverage**: 77 tests across 4 test suites (homepage, user-journeys, header.responsive, footer.responsive) - **Visual regression separation**: Dedicated `visual-regression.spec.ts` with 23 screenshot tests - **Accessibility testing**: ARIA role validation and screen reader compatibility ### 🎨 Visual Regression Testing - **Deterministic rendering**: Timezone (UTC), locale (en-US), font rasterization settings - **Layout stabilization**: `settle()` helper function for consistent screenshot timing - **CSS normalization**: Deterministic CSS for caret, scrollbars, and font rendering - **Browser-specific snapshots**: Separate baseline images for each browser engine - **Tolerance management**: Configurable pixel difference thresholds for CI environment variations ### ⚡ Performance Testing - **Lighthouse CI integration**: Performance budgets and Core Web Vitals monitoring - **Chrome path resolution**: Robust browser installation and path detection for CI - **Architecture compatibility**: Fixed Node.js x64 vs Chrome arm64 mismatch on macOS runners - **Performance budgets**: Configurable thresholds for TTFB, FCP, LCP, and interaction metrics ### 🔄 CI/CD Pipeline (Gitea Actions) - **7 parallel jobs**: Unit tests, E2E tests, Visual regression, Performance, Storybook, Lint, and Build - **Self-hosted runner**: Optimized for macOS arm64 architecture - **Single-step server management**: Start+test+teardown approach preventing process termination issues - **Port conflict resolution**: Dedicated port 3010 for CI testing, avoiding conflicts with local development - **Gitea compatibility**: Conditional GitHub vs non-GitHub configuration for caching and setup - **Baseline snapshot generation**: `seed-vr-snapshots` job on main branch for stable visual regression baselines ### 🧹 Code Quality & Maintenance - **ESLint configuration**: Comprehensive linting rules with Storybook integration - **Prettier formatting**: Consistent code style across all test files - **TypeScript support**: Full type checking for test files and configuration - **Test documentation**: Comprehensive `docs/TESTING.md` with examples and workflows ## Screenshots <img width="909" alt="Screenshot 2025-09-03 at 12.49.45 PM.png" src="attachments/3eb2ecb6-baa3-4833-91fc-c7f3fdf3bf6c"> ## How to Test ### 🚀 Quick Start ```bash # Install dependencies and Playwright browsers npm install npx playwright install # Run all test suites npm test # Unit tests with coverage npm run e2e # E2E tests across all browsers npm run test:sb # Storybook tests npm run lhci # Performance tests ``` ### 🧪 Comprehensive Testing ```bash # Unit testing npm run test:watch # Watch mode for development npm run test:ui # UI-based test runner # E2E testing npm run e2e:ui # Playwright UI for debugging npm run e2e:serve # Dev server + E2E tests # Visual regression npm run visual:test # Run visual regression tests npm run visual:update # Update snapshots # Performance monitoring npm run performance:budget # Budget-based testing npm run performance:monitor # Continuous monitoring ``` ### 🔍 Verification Checklist - [ ] **Unit Tests**: 124 tests passing with >85% coverage - [ ] **E2E Tests**: All 77 tests pass across Chromium, Firefox, WebKit, and Mobile - [ ] **Visual Regression**: 23 screenshot tests generate consistent results - [ ] **Performance**: Lighthouse CI scores meet budget thresholds - [ ] **Storybook**: Builds successfully with GitHub Pages configuration - [ ] **Responsive Design**: All breakpoints render correctly (xs through 4xl) - [ ] **Accessibility**: WCAG compliance across all components - [ ] **Cross-browser**: Consistent behavior in Chromium, Firefox, WebKit, and Mobile ### 🌐 Browser Testing Matrix - **Chromium**: Full test suite validation - **Firefox**: Cross-engine compatibility - **WebKit**: Safari engine validation - **Mobile**: Responsive design verification ## Notes ### 🏗️ Architecture Decisions - **Testing separation**: Functional E2E tests separate from visual regression for maintainability - **CI optimization**: Single-step server management prevents environment drift between CI steps - **Port isolation**: Port 3010 dedicated to CI to avoid conflicts with local development on 3000 - **Snapshot strategy**: CI-generated baseline snapshots eliminate local vs CI environment differences ### 🔧 CI Environment - **Self-hosted runner**: Manual activation for resource management and control - **Architecture compatibility**: arm64 Node.js + arm64 Chrome for optimal performance - **Gitea integration**: Conditional configuration for GitHub vs Gitea environments - **Baseline management**: Automated snapshot generation on main branch for stable comparisons ### 🚀 Future Improvements - **Test coverage expansion**: Additional edge cases and accessibility scenarios - **Performance optimization**: Dynamic budget adjustment based on production metrics - **Visual regression stability**: Enhanced deterministic rendering for CI environments - **Storybook enhancement**: Additional addons for component testing and documentation ### �� Known Limitations - **Visual regression variations**: Slight differences between local and CI due to font rendering - **Performance thresholds**: Conservative budgets that may need tuning based on real usage - **Placeholder content**: Some tests use placeholder links that don't navigate to actual pages - **CI runner management**: Manual activation required for resource optimization ### 🔗 Dependencies & Integrations - **Playwright**: E2E testing framework with visual regression capabilities - **Lighthouse CI**: Performance monitoring and budget enforcement - **Storybook**: Component development and testing environment - **Vitest**: Fast unit test runner with React Testing Library integration - **Gitea Actions**: CI/CD platform for automated testing and deployment - **Next.js 15**: Application framework with built-in testing support ### �� Documentation - **Comprehensive testing guide**: `docs/TESTING.md` with examples and workflows - **Runner management scripts**: Shell scripts for CI runner control - **Configuration examples**: Playwright, Storybook, and CI workflow configurations - **Best practices**: Testing patterns and maintenance guidelines

Screenshot 2025-09-03 at 12.49.45 PM.png

180 KiB

an.di added 30 commits 2025-08-30 02:30:34 +00:00

Initial testing framework 165f7f8179

Unit tests for main index components c8d19089a7

E2E Testing setup 382f4a7541

Fix unit tests c1122c8426

Update visual-regression.spec.ts b0943f445c

Gitea actions added c88e69de14

feat: add complete CI pipeline with Gitea Actions runner configuration 787c6cfe6e

fix: update workflow to run on current branch and use macos-latest runner dec2757f88

feat: set up persistent runner service and proper PR workflow 9d7e0095f2

docs: add manual runner management workflow to testing documentation f87bff4675

Update testing runner 9b7890ca64

fix: adjust coverage thresholds to get CI passing 0c5e8d4dd8

fix: simplify runner config to use host mode only 09603d1b54

Add more unit tests 54227d1930

Improved E2E testing f676f8ec24

Added accessibility tests 705cfed075

Accessibility test fixed e8691efac4

Visual regression tests refined f7621d2086

Performance testing implemented d300772f19

Fix vitest config 4968baafbb

EMS update 2e010e30b5

Integration tests added 2667d054e2

Fix Gitea compatability issues f7f76a2396

Snapshot management 6f7baa6f2f

Add baseline visual regression snapshots for all browser projects (chromium, firefox, webkit, mobile) - 116 snapshots total 64cccac9e7

Fix performance tests: Install @lhci/cli, add Lighthouse CI configuration, and update CI workflow for Gitea compatibility e0d948d24c

Fix Storybook build: Update imports to @storybook/test and add interactions addon 531389d65a

Consolidate storybook configuration c204eabbe5

Fix ESLint TypeScript parser issue: Add TypeScript dependencies and tsconfig.json 5ba0c3c49c

Fix Storybook build: Resolve version conflicts and add required addons 4b50fba89b

an.di added 1 commit 2025-08-30 02:30:44 +00:00

Fix label mismatch: Change all jobs to use [self-hosted, macos-latest] to match runner labels 6994f7508f

an.di added 1 commit 2025-08-30 02:44:37 +00:00

Fix runner issues: Correct labels, fix YAML indentation, add canary job 9ecba9a6f1

an.di added 2 commits 2025-08-30 03:14:17 +00:00

Update workflow to match runner labels: Use self-hosted:host and macos-latest:host 73798b588a

Runner update 25c262e6eb

an.di added 1 commit 2025-08-30 03:25:38 +00:00

Update runner config and workflow to use host mode c8f72e6ec6

an.di added 1 commit 2025-08-30 03:33:42 +00:00

Create .runner.pid 62b1fd3d4a

an.di added 1 commit 2025-08-30 03:38:07 +00:00

Fix duplicate workflow runs: Use Option A - run CI once for PRs, only push to protected branches 42a2fda6eb

an.di added 1 commit 2025-08-30 03:42:34 +00:00

Fix label matching: Use clean label names in runs-on, keep execution mode in runner config 0b7b8df05b

an.di added 1 commit 2025-08-30 03:43:01 +00:00

Update .runner.pid 7123cde0e5

an.di force-pushed adilallo/enhancement/TestingFramework2 from 7123cde0e5 to 8965dd5878 2025-08-30 03:50:17 +00:00 Compare

an.di added 1 commit 2025-08-30 03:52:18 +00:00

Fix duplicate triggers: Only run CI on PRs to main/develop, not feature branch pushes bead0c7373

an.di force-pushed adilallo/enhancement/TestingFramework2 from bead0c7373 to cfe54737b8 2025-08-30 03:55:21 +00:00 Compare

an.di added 9 commits 2025-08-30 04:03:40 +00:00

Fix runner issues: Correct labels, fix YAML indentation, add canary job 9ecba9a6f1

Update workflow to match runner labels: Use self-hosted:host and macos-latest:host 73798b588a

Runner update 25c262e6eb

Update runner config and workflow to use host mode c8f72e6ec6

Create .runner.pid 62b1fd3d4a

Test workflow trigger from working state 8965dd5878

Fix duplicate triggers: Only run CI on PRs to main/develop, not feature branch pushes bead0c7373

Test new runner registration b76d62deb5

Update runner 908fc79303

an.di added 2 commits 2025-08-30 04:06:28 +00:00

Test runner with container config disabled 9133ff8697

Update runner config 0e08e838e8

an.di added 1 commit 2025-08-30 04:10:43 +00:00

Test runner 8e0b4246b2

an.di added 1 commit 2025-08-30 04:22:04 +00:00

Fix test issues: Use getAllByRole for multiple elements, add React cleanup e44d7c1b92

an.di added 1 commit 2025-08-30 04:35:30 +00:00

Fix axe-core imports: Use AxeBuilder API for newer version afdf7ce595

an.di added 1 commit 2025-08-30 04:44:41 +00:00

Fix E2E tests: Use npm run start instead of preview to avoid double build 8969ead3bf

an.di added 1 commit 2025-08-30 05:06:45 +00:00

Fix server startup: Use explicit host binding and verification for all jobs 89f1ee328f

an.di added 1 commit 2025-08-30 05:17:36 +00:00

Implement robust server startup with comprehensive debugging for all jobs 7b660e41eb

an.di added 1 commit 2025-08-30 05:29:35 +00:00

Fix E2E job: Implement robust server startup with proper health checks and cleanup 92f67dc678

an.di added 2 commits 2025-08-30 17:59:50 +00:00

Apply robust server startup pattern to all jobs (visual-regression, performance) with proper cleanup 8efe237018

Update ci.yaml bb483c6139

an.di added 2 commits 2025-08-30 18:03:21 +00:00

Revert "Update ci.yaml" ... 41a1fa84d7

This reverts commit bb483c6139.

Revert "Apply robust server startup pattern to all jobs (visual-regression, performance) with proper cleanup" ... de04405de7

This reverts commit 8efe237018.

an.di added 4 commits 2025-08-30 19:48:11 +00:00

Fix and improve basline tests c20f704ccf

Fix visual regression and performance server issue 1e795e1340

Fix prettier formatting issues 12deae75e8

Remove canary test 494fd9cca1

an.di added 1 commit 2025-08-31 04:34:57 +00:00

Fix performance runner test eb407e03ee

an.di added 2 commits 2025-09-02 23:38:20 +00:00

Fix Storybook runner tests 394173161c

Fix failing E2E tests edf8637d7d

an.di added 1 commit 2025-09-03 01:07:07 +00:00

Fix playwright config mismatch 1cda0e7ad3

an.di added 1 commit 2025-09-03 01:27:07 +00:00

Change runner to use dev server e6bde95e6f

an.di added 1 commit 2025-09-03 02:58:10 +00:00

Add server debugging e071449ce9

an.di added 1 commit 2025-09-03 03:28:38 +00:00

Update ci.yaml 1253779c00

an.di added 1 commit 2025-09-03 03:40:29 +00:00

Try single step approach fd14667da2

an.di added 1 commit 2025-09-03 03:42:02 +00:00

Fix indentation issue 4ba5075a9f

an.di added 1 commit 2025-09-03 03:46:32 +00:00

Fix more indentation and syntax errors 4a8f99a907

an.di added 1 commit 2025-09-03 03:58:03 +00:00

Seed snapshots on branch c2de9e4788

an.di added 1 commit 2025-09-03 04:04:36 +00:00

Seed screenshots for E2E tests 07faeb2460

an.di added 1 commit 2025-09-03 04:43:48 +00:00

Update playwright snapshot names ce4a5efdda

an.di added 1 commit 2025-09-03 04:52:40 +00:00

Fix visual regression test snapshot naming and add missing snapshots 147dd013ab

an.di force-pushed adilallo/enhancement/TestingFramework2 from 147dd013ab to a2381fe148 2025-09-03 05:29:40 +00:00 Compare

an.di added 1 commit 2025-09-03 14:54:55 +00:00

Fix visual regression tests fae2b57c4f

an.di added 3 commits 2025-09-03 15:33:27 +00:00

Resolve pixel drift d7495d8f31

Attempt to fix performance runner 1e299af234

Run Prettier 6b1bed3395

an.di added 2 commits 2025-09-03 16:21:51 +00:00

Fix chromium and webkit tests 0fc5bf30f6

Fix performance test: use mac_arm platform and ensure arm64 Node for Lighthouse 2fcc360e7d

an.di added 4 commits 2025-09-03 17:03:39 +00:00

Attempt to fix snapshot issue 8ab65d8c3c

Fix Chrome path resolution in performance test step 24c8fc525e

Implement comprehensive visual regression stability improvements 5a7295ff5d

Run lint and prettier 1e16c8b6ff

an.di added 1 commit 2025-09-03 17:28:08 +00:00

Fix cache playwrite and performance job 704bc01e23

an.di added 1 commit 2025-09-03 17:42:18 +00:00

Attempt to fix playwright key and performance issue 3bb4139aeb

an.di added 1 commit 2025-09-03 17:59:07 +00:00

Attempt to fix playwright and performance again b6446899b1

an.di added 1 commit 2025-09-03 18:18:25 +00:00

Remove playwright caching a34cc1d788

an.di changed target branch from main to adilallo/enhancement/TestingFramework 2025-09-03 18:38:53 +00:00

an.di changed target branch from adilallo/enhancement/TestingFramework to main 2025-09-03 18:40:40 +00:00

an.di changed title from fix-runner-trigger to Testing Framwork 2025-09-03 18:42:21 +00:00

an.di self-assigned this 2025-09-03 18:50:16 +00:00

an.di merged commit 50efb6a22a into main 2025-09-03 18:50:40 +00:00

an.di referenced this issue from a commit 2025-09-03 18:50:41 +00:00

Merge pull request 'Testing Framwork' (#17) from adilallo/enhancement/TestingFramework2 into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

an.di (Andy DiLallo) ntnsndr (Nathan Schneider)

No Assignees an.di

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: CommunityRule/community-rule#17