Skip to content

Conversation

@ejsmith
Copy link
Contributor

@ejsmith ejsmith commented Jun 28, 2025

🔧 Add Resilience Policy Provider Implementation

Overview

This PR introduces a new resilience policy provider implementation to Foundatio that allows end users to customize and replace how retries and resilience patterns are handled throughout the library. This provides a lightweight, high-performance alternative to external libraries like Polly while maintaining compatibility with existing resilience solutions.

Features Added

Core Implementation

  • ResiliencePolicyProvider: Central provider for managing named resilience policies
  • ResiliencePolicy: Lightweight, configurable policy with retry logic and circuit breaker support
  • CircuitBreaker: Built-in circuit breaker implementation with configurable failure thresholds
  • Builder Pattern: Fluent API for easy policy configuration

Key Capabilities

  • Configurable Retries: Support for exponential backoff, linear delays, and custom retry strategies
  • Circuit Breaker: Automatic failure detection with half-open state testing
  • Named Policies: Multiple policies per application with easy retrieval
  • Exception Filtering: Configurable exception types to handle or ignore
  • Timeout Support: Per-operation timeouts with cancellation
  • Jitter: Optional randomization to prevent thundering herd problems
  • Integration Ready: Designed to work seamlessly with existing Foundatio components

Usage Examples

Basic Policy Setup

// Simple default policy
var provider = new ResiliencePolicyProvider()
    .WithDefaultPolicy(p => p
        .WithMaxAttempts(3)
        .WithExponentialDelay(TimeSpan.FromSeconds(1))
        .WithJitter());

// Get and use the policy
var policy = provider.GetPolicy();
await policy.ExecuteAsync(async ct => 
{
    // Your operation here
    await SomeUnreliableOperationAsync(ct);
});

Named Policies for Different Scenarios

var provider = new ResiliencePolicyProvider()
    .WithPolicy("database", p => p
        .WithMaxAttempts(5)
        .WithLinearDelay(TimeSpan.FromSeconds(2))
        .WithTimeout(TimeSpan.FromSeconds(30))
        .WithUnhandledException<InvalidOperationException>())
    .WithPolicy<MyService>(p => p
        .WithMaxAttempts(3)
        .WithExponentialDelay(TimeSpan.FromSeconds(1))
        .WithCircuitBreaker(cb => cb
            .WithFailureRatio(0.5)
            .WithMinimumCalls(10)
            .WithBreakDuration(TimeSpan.FromSeconds(30))));

// Use specific policies
var dbPolicy = provider.GetPolicy("database");
var apiPolicy = provider.GetPolicy<MyService>();

Circuit Breaker Configuration

var policy = new ResiliencePolicyBuilder()
    .WithCircuitBreaker(cb => cb
        .WithSamplingDuration(TimeSpan.FromSeconds(60))
        .WithFailureRatio(0.1) // 10% failure rate
        .WithMinimumCalls(100) // At least 100 calls before opening
        .WithBreakDuration(TimeSpan.FromSeconds(30))); // Stay open for 30 seconds

Custom Retry Logic

var policy = new ResiliencePolicyBuilder()
    .WithShouldRetry((attempt, exception) => 
    {
        // Custom logic for determining if we should retry
        return exception is HttpRequestException && attempt <= 3;
    })
    .WithGetDelay(attempt => 
    {
        // Custom delay calculation
        return TimeSpan.FromMilliseconds(Math.Pow(2, attempt) * 100);
    });

Integration with Foundatio Components

The resilience policies integrate seamlessly with existing Foundatio components through the IHaveResiliencePolicyProvider interface:

// Example with a hypothetical service
public class MyService
{
    private readonly IResiliencePolicy _policy;
    
    public MyService(IResiliencePolicyProvider resiliencePolicyProvider)
    {
        _policy = _resiliencePolicyProvider.GetPolicy<MyService>();
    }
    
    public async Task DoWorkAsync()
    {
        await _policy.ExecuteAsync(async ct => 
        {
            // Resilient operation
        });
    }
}

Performance Benchmarks

Our benchmarks show excellent performance characteristics compared to Polly:

Scenario Foundatio Polly Performance Gain
No Retries (Async) 603 ns 606 ns ~Same
Standard Config (Async) 567 ns 1,072 ns 47% faster
With Result (Async) 39 ns 134 ns 71% faster
Memory Allocation 810 B 1,416 B 43% less

The Foundatio implementation provides:

  • Significantly better performance for standard retry scenarios
  • 🔋 Lower memory allocation reducing GC pressure
  • 📦 Smaller dependency footprint with zero external dependencies

Breaking Changes

None - This is a purely additive feature that doesn't modify any existing APIs.

Migration from Polly

For users currently using Polly, migration is straightforward:

// Before (Polly)
var policy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(3, retryAttempt => 
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

await policy.ExecuteAsync(async () => await MyOperationAsync());

// After (Foundatio)
var policy = new ResiliencePolicy()
    .WithMaxAttempts(3)
    .WithExponentialDelay(TimeSpan.FromSeconds(1))
    .WithUnhandledException<ArgumentException>(); // Exclude unhandled types

await policy.ExecuteAsync(async ct => await MyOperationAsync());

Integration with External Resilience Libraries

The resilience policy provider design allows you to easily integrate with external libraries like Polly by implementing a custom IResiliencePolicy wrapper:

using Polly;
using Foundatio.Utility;

// Custom Polly wrapper implementing Foundatio's IResiliencePolicy
public class PollyResiliencePolicy : IResiliencePolicy
{
    private readonly IAsyncPolicy _pollyPolicy;

    public PollyResiliencePolicy(IAsyncPolicy pollyPolicy)
    {
        _pollyPolicy = pollyPolicy;
    }

    public async Task ExecuteAsync(Func<CancellationToken, Task> operation, CancellationToken cancellationToken = default)
    {
        await _pollyPolicy.ExecuteAsync(async (ct) => await operation(ct), cancellationToken);
    }

    public async Task<T> ExecuteAsync<T>(Func<CancellationToken, Task<T>> operation, CancellationToken cancellationToken = default)
    {
        return await _pollyPolicy.ExecuteAsync(async (ct) => await operation(ct), cancellationToken);
    }
}

// Custom provider that uses Polly policies
public class PollyResiliencePolicyProvider : IResiliencePolicyProvider
{
    private readonly Dictionary<string, IResiliencePolicy> _policies = new();
    private readonly IResiliencePolicy _defaultPolicy;

    public PollyResiliencePolicyProvider()
    {
        // Create default Polly policy
        var defaultPollyPolicy = Policy
            .Handle<Exception>()
            .WaitAndRetryAsync(3, retryAttempt => 
                TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
        
        _defaultPolicy = new PollyResiliencePolicy(defaultPollyPolicy);

        // Create named Polly policies
        var dbPollyPolicy = Policy
            .Handle<SqlException>()
            .Or<TimeoutException>()
            .WaitAndRetryAsync(5, retryAttempt => 
                TimeSpan.FromSeconds(retryAttempt * 2));
        
        _policies["database"] = new PollyResiliencePolicy(dbPollyPolicy);

        // Polly circuit breaker example
        var apiPollyPolicy = Policy
            .Handle<HttpRequestException>()
            .CircuitBreakerAsync(
                handledEventsAllowedBeforeBreaking: 5,
                durationOfBreak: TimeSpan.FromSeconds(30));
        
        _policies["external-api"] = new PollyResiliencePolicy(apiPollyPolicy);
    }

    public IResiliencePolicy GetPolicy(string name = null)
    {
        if (string.IsNullOrEmpty(name))
            return _defaultPolicy;

        return _policies.TryGetValue(name, out var policy) ? policy : _defaultPolicy;
    }
}

// Register the Polly-based provider with dependency injection
services.AddSingleton<IResiliencePolicyProvider, PollyResiliencePolicyProvider>();

// Now all Foundatio components will use Polly policies
var cache = new InMemoryCacheClient(new InMemoryMessageBus(), provider);
var queue = new InMemoryQueue<MyWorkItem>(provider);
// etc.

This approach allows you to:

  • ✅ Use Polly's advanced features (like sophisticated circuit breakers, bulkhead isolation, etc.)
  • ✅ Leverage existing Polly policies and configurations
  • ✅ Maintain compatibility with Foundatio's resilience interfaces
  • ✅ Easily switch between Foundatio's built-in resilience and Polly
  • ✅ Gradually migrate from Polly to Foundatio's implementation or vice versa

Advanced Features

Manual Circuit Breaker Control

var circuitBreaker = new CircuitBreaker();

// Manually open the circuit
circuitBreaker.Open();

// Check circuit state
if (circuitBreaker.State == CircuitState.Closed)
{
    // Safe to proceed
}

// Manually close the circuit
circuitBreaker.Close();

Custom Exception Handling

var policy = new ResiliencePolicyBuilder()
    .WithShouldRetry((attempt, ex) => 
    {
        // Retry for HTTP errors but not for auth errors
        return ex is HttpRequestException && 
               !ex.Message.Contains("401") && 
               attempt <= 3;
    });

Integration Points

The resilience policy provider is designed to integrate with:

  • 🔄 Messaging components for reliable message processing
  • 💾 Storage operations for transient failure handling
  • 🌐 HTTP clients for robust API communications
  • Background jobs for reliable task execution
  • 📊 Metrics collection for failure tracking

Testing

The implementation includes comprehensive unit tests covering:

  • ✅ Basic retry functionality
  • ✅ Circuit breaker state transitions
  • ✅ Exception filtering
  • ✅ Timeout handling
  • ✅ Jitter behavior
  • ✅ Performance characteristics

Documentation Updates

  • ✅ Updated README.md with resilience policy information
  • ✅ Added comprehensive usage examples
  • ✅ Performance comparison with alternatives
  • ✅ Migration guide for existing users

This resilience policy implementation provides Foundatio users with a high-performance, lightweight alternative for handling transient failures while maintaining the option to integrate with external resilience libraries when needed.

This comment was marked as outdated.

@ejsmith ejsmith changed the title Add extensible resilience pipelines Add extensible resilience policies Jun 29, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces extensible resilience policies to Foundatio including a new ResiliencePolicyProvider, a ResiliencePolicy with fluent builder APIs, and updates across caching, queues, locks, messaging, storage, and jobs to integrate with the new resilience system.

  • Added core resilience types and builder patterns for configuring retries, circuit breakers, timeouts, jitter, and custom exception filtering.
  • Updated tests, benchmarks, and documentation to demonstrate and validate the new resilience functionality.

Reviewed Changes

Copilot reviewed 47 out of 49 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/Foundatio.Tests/Utility/RunTests.cs Removed obsolete tests for Run utility methods.
tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs Updated tests to validate the new resilience policies.
tests/Foundatio.Tests/Utility/PollyResiliencePolicyProvider.cs Added tests for the Polly-based policy wrapper integration.
tests/Foundatio.Tests/Queue/InMemoryQueueTests.cs Increased timeout duration for queue tests to support resilience.
tests/Foundatio.Tests/Locks/InMemoryLockTests.cs Updated lock provider constructor signature to include resilience.
... Numerous updates in caching, storage, messaging, jobs and benchmarks to use the new resilience interfaces.
README.md Updated documentation with new Resilience section and samples.
benchmarks/* Added benchmarks to compare performance of ResiliencePolicy vs Polly.

@ejsmith ejsmith merged commit 080149a into main Jul 1, 2025
3 checks passed
@ejsmith ejsmith deleted the resilience-provider branch July 1, 2025 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants