Skip to main content

Testing Strategies & Requirements

SpecifyX employs a multi-layered testing strategy to ensure code quality, reliability, and performance. This guide covers all testing approaches, requirements, and best practices for contributors.

Testing Architecture Overview​

Test Organization Structure​

tests/
├── __init__.py
├── contract/              # Abstract base class contract tests
│   ├── test_template_service.py
│   ├── test_config_service.py
│   └── test_project_manager.py
├── unit/                  # Isolated component tests
│   ├── test_ui_utilities.py
│   ├── test_script_helpers.py
│   └── test_ui_helpers.py
├── integration/           # Cross-component workflow tests
│   ├── test_project_init.py
│   ├── test_template_rendering.py
│   ├── test_config_loading.py
│   └── test_cross_platform.py
└── performance/           # Benchmark and load tests
    ├── test_template_performance.py
    └── test_cli_startup_time.py

Testing Philosophy​

  1. Contract-First Testing - Test abstract interfaces before implementations
  2. Behavior-Driven - Test what the code should do, not how it does it
  3. Isolation - Each test should be independent and repeatable
  4. Real-World Scenarios - Integration tests mirror actual usage patterns
  5. Performance Awareness - Critical paths have performance benchmarks

Contract Testing​

Contract tests ensure that all implementations of abstract base classes behave consistently and correctly.

Template Service Contract Tests​

class TestTemplateService:
    """Contract tests for TemplateService implementations"""
    
    @pytest.fixture
    def service(self) -> TemplateService:
        """Override in concrete test classes"""
        raise NotImplementedError("Concrete test classes must provide service fixture")
    
    def test_render_template_with_valid_inputs(self, service: TemplateService):
        """Test successful template rendering"""
        context = TemplateContext(
            project_name="test-project",
            ai_assistant="claude",
            branch_naming_config=BranchNamingConfig(),
        )
        
        # This test ensures all implementations can render basic templates
        result = service.render_template("basic-template", context)
        
        assert isinstance(result, str)
        assert len(result) > 0
        assert "test-project" in result
    
    def test_render_template_with_missing_template(self, service: TemplateService):
        """Test error handling for missing templates"""
        context = TemplateContext(
            project_name="test-project", 
            ai_assistant="claude"
        )
        
        with pytest.raises(FileNotFoundError, match="Template not found"):
            service.render_template("nonexistent-template", context)
    
    def test_validate_template_syntax_with_valid_template(self, service: TemplateService, tmp_path):
        """Test template syntax validation with valid Jinja2"""
        template_file = tmp_path / "valid.j2"
        template_file.write_text("Hello {{ project_name }}!")
        
        is_valid, error_msg = service.validate_template_syntax(template_file)
        
        assert is_valid is True
        assert error_msg is None
    
    def test_validate_template_syntax_with_invalid_template(self, service: TemplateService, tmp_path):
        """Test template syntax validation with invalid Jinja2"""
        template_file = tmp_path / "invalid.j2"
        template_file.write_text("Hello {{ unclosed_variable")
        
        is_valid, error_msg = service.validate_template_syntax(template_file)
        
        assert is_valid is False
        assert error_msg is not None
        assert "syntax error" in error_msg.lower()


# Concrete implementation test
class TestJinjaTemplateService(TestTemplateService):
    """Contract tests for JinjaTemplateService"""
    
    @pytest.fixture
    def service(self) -> JinjaTemplateService:
        return JinjaTemplateService()
    
    # Additional implementation-specific tests
    def test_jinja_specific_features(self, service: JinjaTemplateService):
        """Test Jinja2-specific functionality"""
        # Test custom filters, environments, etc.
        pass

Contract Test Benefits​

  • Interface Compliance - Ensures all implementations follow the same contract
  • Behavioral Consistency - Guarantees consistent behavior across implementations
  • Documentation - Contract tests serve as executable specifications
  • Refactoring Safety - Changing implementations won't break expected behavior

Unit Testing​

Unit tests focus on isolated components with minimal dependencies.

Testing Utilities and Helpers​

class TestUIHelpers:
    """Unit tests for UI utility functions"""
    
    def test_format_progress_message(self):
        """Test progress message formatting"""
        message = format_progress_message("Processing", 3, 10)
        assert "Processing (3/10)" in message
        assert "30%" in message
    
    def test_format_progress_message_with_zero_total(self):
        """Test progress message with edge case"""
        message = format_progress_message("Processing", 0, 0)
        assert "Processing (0/0)" in message
        assert "100%" in message
    
    @pytest.mark.parametrize("current,total,expected_percentage", [
        (0, 10, "0%"),
        (5, 10, "50%"),
        (10, 10, "100%"),
        (7, 10, "70%"),
    ])
    def test_calculate_percentage(self, current, total, expected_percentage):
        """Test percentage calculation with various inputs"""
        result = calculate_percentage(current, total)
        assert result == expected_percentage


class TestFileOperations:
    """Unit tests for file operation utilities"""
    
    def test_safe_file_write(self, tmp_path):
        """Test atomic file writing"""
        target_file = tmp_path / "test.txt"
        content = "Hello, World!"
        
        safe_file_write(target_file, content)
        
        assert target_file.exists()
        assert target_file.read_text() == content
    
    def test_safe_file_write_preserves_existing_on_error(self, tmp_path, monkeypatch):
        """Test that existing files are preserved when writes fail"""
        target_file = tmp_path / "existing.txt"
        target_file.write_text("Original content")
        
        # Mock write failure
        monkeypatch.setattr("pathlib.Path.write_text", lambda self, content: (_ for _ in ()).throw(OSError("Disk full")))
        
        with pytest.raises(OSError):
            safe_file_write(target_file, "New content")
        
        # Original content should be preserved
        assert target_file.read_text() == "Original content"

Unit Test Guidelines​

  • Single Responsibility - Test one function/method per test
  • No External Dependencies - Mock file systems, networks, databases
  • Fast Execution - Unit tests should run in milliseconds
  • Parametrized Testing - Use @pytest.mark.parametrize for multiple inputs
  • Edge Cases - Test boundary conditions and error scenarios

Integration Testing​

Integration tests verify that components work together correctly in realistic scenarios.

Project Initialization Workflow​

class TestProjectInitialization:
    """Integration tests for complete project initialization"""
    
    def test_complete_project_init_workflow(self, tmp_path):
        """Test full project initialization from start to finish"""
        project_name = "test-project"
        ai_assistant = "claude"
        
        # Initialize project manager
        manager = ProjectManager()
        
        # Execute initialization
        result = manager.initialize_project(
            project_name=project_name,
            target_directory=tmp_path,
            ai_assistant=ai_assistant,
            branch_pattern="feature/{feature-name}"
        )
        
        # Verify successful initialization
        assert result.success is True
        assert result.project_path.exists()
        
        # Verify directory structure
        project_dir = tmp_path / project_name
        assert (project_dir / ".specify").exists()
        assert (project_dir / ".claude" / "commands").exists()
        assert (project_dir / ".specify" / "scripts").exists()
        
        # Verify configuration files
        config_file = project_dir / ".specify" / "config.toml"
        assert config_file.exists()
        
        # Verify configuration content
        config_data = toml.load(config_file)
        assert config_data["project"]["name"] == project_name
        assert config_data["project"]["template_settings"]["ai_assistant"] == ai_assistant
        
        # Verify generated scripts
        scripts_dir = project_dir / ".specify" / "scripts"
        assert len(list(scripts_dir.glob("*.py"))) > 0
        
        # Verify AI-specific commands
        commands_dir = project_dir / ".claude" / "commands"
        assert len(list(commands_dir.glob("*.md"))) > 0
    
    def test_project_init_with_existing_directory(self, tmp_path):
        """Test initialization behavior with existing directory"""
        project_name = "existing-project"
        project_dir = tmp_path / project_name
        project_dir.mkdir()
        
        # Create conflicting file
        (project_dir / ".specify").mkdir()
        (project_dir / ".specify" / "config.toml").write_text("existing content")
        
        manager = ProjectManager()
        
        # Should handle existing directory gracefully
        result = manager.initialize_project(
            project_name=project_name,
            target_directory=tmp_path,
            ai_assistant="claude",
            overwrite_existing=False
        )
        
        # Verify appropriate handling (implementation-dependent)
        assert result.success is False or result.warnings is not None


class TestTemplateRendering:
    """Integration tests for template rendering workflows"""
    
    def test_ai_aware_template_rendering(self, tmp_path):
        """Test rendering of AI-aware templates with different assistants"""
        template_service = JinjaTemplateService()
        
        for ai_assistant in ["claude", "gpt", "gemini"]:
            context = TemplateContext(
                project_name="ai-test-project",
                ai_assistant=ai_assistant,
                project_path=tmp_path,
                branch_naming_config=BranchNamingConfig(),
            )
            
            # Render AI-specific template
            result = template_service.render_template("ai-aware-command", context)
            
            # Verify AI-specific content
            assert ai_assistant in result
            assert "ai-test-project" in result
            
            # Verify platform compatibility
            if platform.system() == "Windows":
                assert ".bat" in result or "cmd" in result
            else:
                assert ".sh" in result or "bash" in result
    
    def test_cross_platform_script_generation(self, tmp_path):
        """Test script generation across different platforms"""
        template_service = JinjaTemplateService()
        
        context = TemplateContext(
            project_name="cross-platform-test",
            ai_assistant="claude",
            project_path=tmp_path,
        )
        
        # Mock different platforms
        original_system = platform.system
        
        for platform_name in ["Windows", "Darwin", "Linux"]:
            with patch("platform.system", return_value=platform_name):
                enhanced_context = template_service.enhance_context_with_platform_info(context, platform_name)
                result = template_service.render_template("platform-script", enhanced_context)
                
                # Verify platform-specific content
                if platform_name == "Windows":
                    assert "Windows" in result or ".bat" in result
                else:
                    assert "Unix" in result or ".sh" in result

Configuration Loading and Persistence​

class TestConfigurationManagement:
    """Integration tests for configuration loading and persistence"""
    
    def test_config_roundtrip_persistence(self, tmp_path):
        """Test configuration save and load roundtrip"""
        config_service = ConfigService()
        
        # Create test configuration
        original_config = ProjectConfig(
            name="roundtrip-test",
            branch_naming=BranchNamingConfig(
                patterns=["feature/{feature-name}", "hotfix/{bug-id}"],
                default_pattern="feature/{feature-name}"
            ),
            template_settings=TemplateConfig(
                ai_assistant="claude",
                template_cache_enabled=True,
                template_variables={"author": "Test User", "version": "1.0.0"}
            ),
        )
        
        # Save configuration
        config_path = tmp_path / "config.toml"
        success = config_service.save_config(original_config, config_path)
        assert success is True
        assert config_path.exists()
        
        # Load configuration
        loaded_config = config_service.load_config(config_path)
        assert loaded_config is not None
        
        # Verify roundtrip accuracy
        assert loaded_config.name == original_config.name
        assert loaded_config.branch_naming.patterns == original_config.branch_naming.patterns
        assert loaded_config.template_settings.ai_assistant == original_config.template_settings.ai_assistant
        assert loaded_config.template_settings.template_variables == original_config.template_settings.template_variables
    
    def test_config_migration_from_legacy_format(self, tmp_path):
        """Test migration from older configuration formats"""
        # Create legacy format configuration
        legacy_config = tmp_path / "legacy-config.toml"
        legacy_config.write_text("""
        [project]
        name = "legacy-project"
        ai_assistant = "claude"  # Old format
        
        [branch_naming]
        pattern = "feature/{feature-name}"  # Old single pattern format
        """)
        
        config_service = ConfigService()
        loaded_config = config_service.load_config(legacy_config)
        
        # Verify migration to new format
        assert loaded_config is not None
        assert loaded_config.name == "legacy-project"
        assert loaded_config.template_settings.ai_assistant == "claude"
        assert "feature/{feature-name}" in loaded_config.branch_naming.patterns

Performance Testing​

Performance tests ensure that critical operations meet performance requirements.

Template Rendering Benchmarks​

class TestTemplatePerformance:
    """Performance benchmarks for template operations"""
    
    def test_template_rendering_performance(self, benchmark):
        """Benchmark template rendering speed"""
        template_service = JinjaTemplateService()
        
        context = TemplateContext(
            project_name="performance-test",
            ai_assistant="claude",
            branch_naming_config=BranchNamingConfig(),
        )
        
        # Benchmark single template rendering
        result = benchmark(template_service.render_template, "basic-template", context)
        
        # Verify performance requirements
        assert len(result) > 0
        # Template rendering should complete within 100ms
        assert benchmark.stats.mean < 0.1
    
    def test_bulk_template_rendering_performance(self, benchmark):
        """Benchmark rendering multiple templates"""
        template_service = JinjaTemplateService()
        
        context = TemplateContext(
            project_name="bulk-performance-test",
            ai_assistant="claude",
            branch_naming_config=BranchNamingConfig(),
        )
        
        def render_multiple_templates():
            templates = ["basic-template", "script-template", "command-template"]
            results = []
            for template_name in templates:
                result = template_service.render_template(template_name, context)
                results.append(result)
            return results
        
        # Benchmark bulk rendering
        results = benchmark(render_multiple_templates)
        
        # Verify all templates rendered
        assert len(results) == 3
        assert all(len(result) > 0 for result in results)
        
        # Bulk rendering should complete within 300ms
        assert benchmark.stats.mean < 0.3
    
    @pytest.mark.parametrize("template_count", [1, 5, 10, 25])
    def test_template_discovery_scaling(self, template_count, benchmark):
        """Test template discovery performance with varying template counts"""
        template_service = JinjaTemplateService()
        
        def discover_templates():
            return template_service.discover_templates()
        
        templates = benchmark(discover_templates)
        
        # Verify discovery completed
        assert isinstance(templates, list)
        assert len(templates) >= 0
        
        # Discovery should scale linearly and complete quickly
        assert benchmark.stats.mean < 0.05  # 50ms max


class TestCLIStartupPerformance:
    """Performance tests for CLI startup time"""
    
    def test_cli_cold_start_time(self, benchmark):
        """Benchmark CLI cold start performance"""
        def run_help_command():
            result = subprocess.run(
                ["python", "-m", "specify_cli", "--help"],
                capture_output=True,
                text=True,
                timeout=5.0
            )
            return result.returncode
        
        exit_code = benchmark(run_help_command)
        
        # Verify successful execution
        assert exit_code == 0
        
        # CLI should start within 200ms
        assert benchmark.stats.mean < 0.2

Test Data Management​

Fixtures and Test Data​

@pytest.fixture
def sample_project_config() -> ProjectConfig:
    """Provide sample project configuration for testing"""
    return ProjectConfig(
        name="test-project",
        branch_naming=BranchNamingConfig(
            patterns=["feature/{feature-name}", "hotfix/{bug-id}"],
            default_pattern="feature/{feature-name}"
        ),
        template_settings=TemplateConfig(
            ai_assistant="claude",
            config_directory=".specify",
            template_cache_enabled=True,
        ),
    )

@pytest.fixture
def sample_template_context(tmp_path) -> TemplateContext:
    """Provide sample template context for testing"""
    return TemplateContext(
        project_name="fixture-test-project",
        ai_assistant="claude",
        project_path=tmp_path,
        branch_naming_config=BranchNamingConfig(),
        config_directory=".specify",
        creation_date="2024-01-01",
    )

@pytest.fixture
def mock_github_api(monkeypatch):
    """Mock GitHub API responses for testing"""
    def mock_get(url, **kwargs):
        response = Mock()
        response.status_code = 200
        response.json.return_value = {
            "name": "test-template",
            "download_url": "https://example.com/template.zip"
        }
        return response
    
    monkeypatch.setattr("httpx.get", mock_get)
    return mock_get

@pytest.fixture(scope="session")
def template_test_data():
    """Provide test template data for entire test session"""
    return {
        "basic_template": "Hello {{ project_name }}!",
        "ai_aware_template": """
        {% if ai_assistant == "claude" %}
        # Claude-specific content
        {% elif ai_assistant == "gpt" %}
        # GPT-specific content
        {% else %}
        # Generic content
        {% endif %}
        """,
        "platform_template": """
        {% if is_windows %}
        @echo off
        echo "Windows script"
        {% else %}
        #!/bin/bash
        echo "Unix script"
        {% endif %}
        """
    }

Testing Best Practices​

Test Organization​

  1. One Test Class Per Component - Group related tests logically
  2. Descriptive Test Names - Test names should describe the scenario
  3. Setup and Teardown - Use fixtures for common setup/cleanup
  4. Test Independence - Each test should run independently
  5. Fast Feedback - Prioritize fast-running tests for development

Assertion Strategies​

# ✅ Good - Specific assertions
assert result.success is True
assert result.error_message is None
assert len(result.rendered_files) == 3
assert "test-project" in rendered_content

# ❌ Bad - Vague assertions
assert result
assert rendered_content

Mock Usage Guidelines​

# ✅ Good - Mock external dependencies
@patch("httpx.get")
def test_download_template(mock_get):
    mock_get.return_value.status_code = 200
    # Test implementation...

# ✅ Good - Mock file system for isolation
def test_config_save(monkeypatch, tmp_path):
    monkeypatch.setattr("specify_cli.utils.file_operations.get_config_dir", lambda: tmp_path)
    # Test implementation...

# ❌ Bad - Mocking internal logic being tested
@patch("specify_cli.services.template_service.JinjaTemplateService.render_template")
def test_template_service(mock_render):
    # This defeats the purpose of testing the service

Continuous Integration​

Test Execution in CI​

Look at .github/workflows/ci.yml for the CI configuration.

Coverage Requirements​

  • Minimum Coverage: 80% overall code coverage
  • Critical Paths: 95% coverage for core services
  • New Code: 100% coverage for new features
  • Exclude: Test files, CLI entry points, and platform-specific code

By following these testing strategies, you'll contribute robust, reliable code that maintains SpecifyX's high quality standards. Remember: good tests are an investment in the future maintainability and reliability of the codebase.