Testing Framwork #17

Merged
an.di merged 83 commits from adilallo/enhancement/TestingFramework2 into main 2025-09-03 18:50:40 +00:00
7 changed files with 341 additions and 5 deletions
Showing only changes of commit 6f7baa6f2f - Show all commits
+10 -1
View File
@@ -82,7 +82,16 @@ jobs:
- run: npm run preview &
env: { CI: true }
- run: npx wait-on http://localhost:3000
- run: npx playwright test tests/e2e/visual-regression.spec.ts
# Seed snapshots on main branch only (one-time setup)
- name: Seed snapshots (main only)
if: github.ref == 'refs/heads/main'
run: PLAYWRIGHT_UPDATE_SNAPSHOTS=1 npx playwright test tests/e2e/visual-regression.spec.ts --project=chromium
env: { CI: true }
# Run visual regression tests
- name: Run visual regression tests
run: npx playwright test tests/e2e/visual-regression.spec.ts
env: { CI: true }
- name: Package visual artifacts
+110
View File
@@ -0,0 +1,110 @@
# Visual Regression Snapshot Workflow
Quick reference for managing visual regression snapshots.
## 🚀 **First-Time Setup**
```bash
# 1. Generate baseline snapshots (choose one)
npm run seed-snapshots # Docker (recommended for CI consistency)
npm run seed-snapshots:local # Local environment
# 2. Commit the snapshots
git add tests/e2e/visual-regression.spec.ts-snapshots/
git commit -m "Add baseline visual regression snapshots"
# 3. Verify setup
npm run visual:test
```
## 🔄 **Daily Workflow**
```bash
# Run visual regression tests
npm run visual:test
# Run with UI for debugging
npm run visual:ui
# Update snapshots after UI changes
npm run visual:update
```
## 📝 **When UI Changes**
1. **Make your UI changes** (design updates, component modifications, etc.)
2. **Update snapshots to reflect new design:**
```bash
npm run visual:update
```
3. **Review changes:**
```bash
git diff tests/e2e/visual-regression.spec.ts-snapshots/
```
4. **Commit updated snapshots:**
```bash
git add tests/e2e/visual-regression.spec.ts-snapshots/
git commit -m "Update snapshots for [describe changes]"
```
## 🐛 **Troubleshooting**
### **"Snapshot doesn't exist" errors**
- **Cause**: Baseline snapshots haven't been generated
- **Fix**: Run `npm run seed-snapshots:local`
### **Platform differences (macOS vs Linux)**
- **Cause**: Different font rendering between platforms
- **Fix**: Use `npm run seed-snapshots` (Docker container)
### **Minor pixel differences**
- **Cause**: Font rendering, anti-aliasing differences
- **Fix**: Check tolerance settings in `playwright.config.ts`
### **Animation-related failures**
- **Cause**: Animations not fully disabled
- **Fix**: Ensure `animations: "disabled"` is set (already configured)
## 📁 **File Structure**
```
tests/e2e/
├── visual-regression.spec.ts # Test definitions
└── visual-regression.spec.ts-snapshots/ # Baseline images
├── homepage-full-chromium.png
├── homepage-viewport-chromium.png
├── hero-banner-chromium.png
└── ...
```
## ⚡ **Quick Commands Reference**
| Command | Purpose |
| ------------------------------ | ----------------------------------------------- |
| `npm run visual:test` | Run visual regression tests |
| `npm run visual:update` | Update snapshots after UI changes |
| `npm run visual:ui` | Run tests with UI for debugging |
| `npm run seed-snapshots` | Generate baselines with Docker (CI consistency) |
| `npm run seed-snapshots:local` | Generate baselines locally |
## 💡 **Best Practices**
1. **Always review changes** before committing updated snapshots
2. **Use descriptive commit messages** when updating snapshots
3. **Test locally first** before pushing to CI
4. **Use Docker for consistency** when generating baselines
5. **Update snapshots promptly** after UI changes to avoid drift
## 🔗 **Related Documentation**
- [Visual Regression Setup](./VISUAL_REGRESSION_SETUP.md) - Detailed setup guide
- [Testing Strategy](../TESTING_STRATEGY.md) - Overall testing approach
+153
View File
@@ -0,0 +1,153 @@
# Visual Regression Testing Setup
This document explains how to set up and maintain visual regression tests for the CommunityRule platform.
## Overview
Visual regression tests capture screenshots of key UI components and compare them against baseline images to detect unintended visual changes. The tests are configured to work consistently across different environments (local development, CI/CD).
## Configuration
### Playwright Configuration
The visual regression tests are configured in `playwright.config.ts` with the following key settings:
- **OS-agnostic snapshots**: Uses `{testDir}/{testFileName}-snapshots/{arg}-{projectName}.png` template
- **Consistent rendering**: Fixed viewport (1280x800), device scale factor (1), and color scheme (light)
- **Tolerance settings**: Allows 1% pixel difference or 200 pixels maximum difference
- **Animation handling**: Disables animations during screenshot capture
### CI Environment
The CI workflow uses:
- **Ubuntu Linux** with Playwright Docker image for consistent rendering
- **One-time snapshot seeding** on the main branch
- **Artifact packaging** to reduce file count and improve upload performance
## Setting Up Snapshots
### Option 1: CI Seeding (Recommended for New Projects)
1. **Push to main branch**: The CI will automatically generate baseline snapshots
2. **Download artifacts**: Download the `visual-regression-results` artifact
3. **Extract snapshots**: Extract the `tests/e2e/visual-regression.spec.ts-snapshots/` folder
4. **Commit snapshots**: Add and commit the snapshot files to your repository
5. **Remove seeding step**: After snapshots are committed, remove the seeding step from CI
### Option 2: Local Seeding with Docker (Recommended for Existing Projects)
Use the provided script to generate snapshots in the same environment as CI:
```bash
# Using the Docker container (ensures CI consistency)
npm run seed-snapshots
# Or using local environment (may have slight differences)
npm run seed-snapshots:local
```
The Docker approach ensures:
- Same fonts and rendering as CI
- Same file naming conventions
- Same performance characteristics
## Running Visual Regression Tests
### Local Development
```bash
# Run all visual regression tests
npx playwright test tests/e2e/visual-regression.spec.ts
# Run with UI for debugging
npx playwright test tests/e2e/visual-regression.spec.ts --ui
# Update snapshots (use with caution)
PLAYWRIGHT_UPDATE_SNAPSHOTS=1 npx playwright test tests/e2e/visual-regression.spec.ts
```
### CI/CD
Visual regression tests run automatically in the CI pipeline:
- **Main branch**: Generates new snapshots if needed
- **Feature branches**: Compares against existing snapshots
- **Artifacts**: Uploads test results and snapshots for review
## Troubleshooting
### Common Issues
1. **"Snapshot doesn't exist" errors**
- **Cause**: Baseline snapshots haven't been generated
- **Solution**: Run snapshot seeding (see above)
2. **Platform-specific failures**
- **Cause**: Snapshots generated on different OS (macOS vs Linux)
- **Solution**: Use Docker container for local snapshot generation
3. **Minor pixel differences**
- **Cause**: Font rendering differences, anti-aliasing, etc.
- **Solution**: Check tolerance settings in `playwright.config.ts`
4. **Animation-related failures**
- **Cause**: Animations not fully disabled
- **Solution**: Ensure `animations: "disabled"` is set in test configuration
### Updating Snapshots
When intentional UI changes are made:
1. **Local update**:
```bash
PLAYWRIGHT_UPDATE_SNAPSHOTS=1 npx playwright test tests/e2e/visual-regression.spec.ts
```
2. **Review changes**: Check the updated snapshots in `tests/e2e/visual-regression.spec.ts-snapshots/`
3. **Commit changes**: Add and commit the updated snapshot files
4. **Verify**: Run tests again to ensure they pass
### Performance Considerations
- **CI environment**: Tests run slower due to container overhead
- **Local development**: Faster execution with native browser
- **Artifact size**: Snapshots are compressed and packaged to reduce upload time
## Best Practices
1. **Consistent environment**: Always use the same environment for snapshot generation and testing
2. **Meaningful test names**: Use descriptive names for snapshot files
3. **Selective testing**: Test only critical UI components to maintain reasonable test duration
4. **Regular updates**: Update snapshots when making intentional UI changes
5. **Review failures**: Always review visual regression failures before updating snapshots
## File Structure
```
tests/e2e/
├── visual-regression.spec.ts # Test definitions
└── visual-regression.spec.ts-snapshots/ # Baseline images
├── homepage-full-chromium.png
├── homepage-viewport-chromium.png
├── hero-banner-chromium.png
└── ...
```
## Integration with Other Tests
Visual regression tests complement:
- **Unit tests**: Verify component logic
- **Integration tests**: Verify component interactions
- **E2E tests**: Verify user workflows
- **Accessibility tests**: Verify accessibility compliance
Together, these tests provide comprehensive coverage of the application's functionality and appearance.
+6 -1
View File
@@ -25,7 +25,12 @@
"performance:budget": "lhci autorun --budgetPath=performance-budgets.json",
"performance:monitor": "node scripts/performance-monitor.js",
"preview": "next build && next start -p 3000",
"e2e:serve": "start-server-and-test preview http://localhost:3000 e2e"
"e2e:serve": "start-server-and-test preview http://localhost:3000 e2e",
"seed-snapshots": "./scripts/seed-snapshots.sh",
"seed-snapshots:local": "PLAYWRIGHT_UPDATE_SNAPSHOTS=1 npx playwright test tests/e2e/visual-regression.spec.ts --project=chromium",
"visual:test": "npx playwright test tests/e2e/visual-regression.spec.ts",
"visual:update": "PLAYWRIGHT_UPDATE_SNAPSHOTS=1 npx playwright test tests/e2e/visual-regression.spec.ts",
"visual:ui": "npx playwright test tests/e2e/visual-regression.spec.ts --ui"
},
"dependencies": {
"next": "15.2.4",
+14 -1
View File
@@ -3,7 +3,14 @@ import { defineConfig, devices } from "@playwright/test";
export default defineConfig({
testDir: "tests/e2e",
timeout: 60_000,
expect: { timeout: 10_000 },
expect: {
timeout: 10_000,
toHaveScreenshot: {
animations: "disabled",
maxDiffPixelRatio: 0.01, // 1% pixels may differ
maxDiffPixels: 200, // or an absolute pixel count
},
},
fullyParallel: true,
retries: process.env.CI ? 2 : 0,
reporter: [["list"], ["html", { open: "never" }]],
@@ -12,6 +19,9 @@ export default defineConfig({
trace: "on-first-retry",
screenshot: "only-on-failure",
video: "retain-on-failure",
colorScheme: "light", // Ensure consistent color scheme
viewport: { width: 1280, height: 800 }, // Consistent viewport
deviceScaleFactor: 1, // Consistent device scale
},
webServer: {
command: "npm run dev",
@@ -19,6 +29,9 @@ export default defineConfig({
reuseExistingServer: true,
timeout: 120_000,
},
// OS-agnostic snapshot path template (removes platform-specific suffixes)
snapshotPathTemplate:
"{testDir}/{testFileName}-snapshots/{arg}-{projectName}.png",
projects: [
{ name: "chromium", use: { ...devices["Desktop Chrome"] } },
{ name: "firefox", use: { ...devices["Desktop Firefox"] } },
+43
View File
@@ -0,0 +1,43 @@
#!/bin/bash
# Script to seed visual regression snapshots using Playwright Docker container
# This ensures the snapshots are generated in the same environment as CI
set -e
echo "🚀 Seeding visual regression snapshots using Playwright Docker container..."
# Check if Docker is available
if ! command -v docker &> /dev/null; then
echo "❌ Docker is not installed or not available in PATH"
exit 1
fi
# Run the Playwright container and generate snapshots
docker run --rm -it \
-v "$(pwd):/work" \
-w /work \
mcr.microsoft.com/playwright:v1.54.2-jammy \
bash -c "
echo '📦 Installing dependencies...'
npm ci
echo '🎭 Installing Playwright browsers...'
npx playwright install --with-deps
echo '🔨 Building application...'
npm run build
echo '🌐 Starting preview server...'
npm run preview &
sleep 10
echo '📸 Generating snapshots...'
PLAYWRIGHT_UPDATE_SNAPSHOTS=1 npx playwright test tests/e2e/visual-regression.spec.ts --project=chromium
echo '✅ Snapshots generated successfully!'
"
echo "🎉 Snapshot seeding completed!"
echo "📁 Check the generated snapshots in: tests/e2e/visual-regression.spec.ts-snapshots/"
echo "💡 Don't forget to commit the snapshot files to your repository"
+5 -2
View File
@@ -1,7 +1,7 @@
import { test, expect } from "@playwright/test";
import { PlaywrightPerformanceMonitor } from "../performance/performance-monitor.js";
// Performance budgets and thresholds
// Environment-aware performance budgets and thresholds
const PERFORMANCE_BUDGETS = {
// Page load performance
page_load_time: 3000, // 3 seconds
@@ -19,7 +19,7 @@ const PERFORMANCE_BUDGETS = {
// Component performance
component_render_time: 500, // 500ms
interaction_time: 100, // 100ms
scroll_performance: 50, // 50ms
scroll_performance: process.env.CI ? 200 : 50, // Looser in CI (200ms vs 50ms)
// Resource performance
network_request_duration: 1000, // 1 second
@@ -48,6 +48,9 @@ test.describe("Performance Monitoring", () => {
let performanceMonitor: PlaywrightPerformanceMonitor;
test.beforeEach(async ({ page }) => {
// Mark tests as slower in CI environment
if (process.env.CI) test.slow();
performanceMonitor = new PlaywrightPerformanceMonitor(page);
performanceMonitor.setThresholds(PERFORMANCE_BUDGETS);
performanceMonitor.setBaselines(BASELINE_METRICS);