hamburgscleanest/guzzle-advanced-throttle

A Guzzle middleware that throttles requests according to (multiple) defined rules.

It is also possible to define a caching strategy. For example, the response can be read from a cache when exceeding rate limits. The cached value can also be preferred to spare your rate limits (force-cache).

Using wildcards in hostnames is also supported.

Install

Via Composer

composer require hamburgscleanest/guzzle-advanced-throttle

Usage

General use

Let's say you wanted to implement the following rules:

20 requests every 1 seconds

100 requests every 2 minutes

1. First, you have to define the rules in a hamburgscleanest\GuzzleAdvancedThrottle\RequestLimitRuleset:

$rules = new RequestLimitRuleset([
        'https://www.google.com' => [
            [
                'max_requests'     => 20,
                'request_interval' => 1
            ],
            [
                'max_requests'     => 100,
                'request_interval' => 120
            ]
        ]
    ]);

2. Your handler stack might look like this:

 $stack = new HandlerStack();
 $stack->setHandler(new CurlHandler());

3. Push hamburgscleanest\GuzzleAdvancedThrottle\Middleware\ThrottleMiddleware to the stack.

It should always be the first middleware on the stack.

 $throttle = new ThrottleMiddleware($rules);

 // Invoke the middleware
 $stack->push($throttle());
 
 // OR: alternatively call the handle method directly
 $stack->push($throttle->handle());

5. Pass the stack to the client

$client = new Client(['base_uri' => 'https://www.google.com', 'handler' => $stack]);

Either the base_uri has to be the same as the defined host in the rules array or you have to request absolute URLs for the middleware to have an effect.

// relative
$response = $client->get('test');

// absolute
$response = $client->get('https://www.google.com/test');

Caching

Beforehand

Responses with an error status code 4xx or 5xx are not cached (even with force-cache enabled)! Note: Currently, also redirect responses (3xx) are not cached.

Available storage adapters

array (default)

This adapter works out of the box. However, it does not persist anything. This one only works within the same scope. It's set as a default because it doesn't need extra configuration.

The recommended adapter is the laravel one.

laravel (Illuminate/Cache) - recommended

You need to provide a config (Illuminate\Config\Repository) for this adapter.

custom (Implements hamburgscleanest\GuzzleAdvancedThrottle\Cache\Interfaces\StorageInterface)

When you create a new implementation, pass the class name to the RequestLimitRuleset::create method. You'll also need to implement any sort of configuration parsing your instance needs. Please see LaravelAdapter for an example.

Usage

$rules = new RequestLimitRuleset(
    [ ... ], 
    'force-cache', // caching strategy
    MyCustomAdapter::class // storage adapter
    );
    
$throttle = new ThrottleMiddleware($rules);

// Invoke the middleware
$stack->push($throttle());  

Laravel Drivers

General settings

These values can be set for every adapter.

    'cache' => [
        'ttl' => 900, // How long should responses be cached for (in seconds)?
        'allow_empty' => true // When this is set to false, empty responses won't be cached.
    ]

File

    'cache' => [
        'driver'  => 'file',
        'options' => [
            'path' => './cache'
        ],
        ...
    ]

Redis

    'cache' => [
        'driver'  => 'redis',
        'options' => [
            'database' => [
                'cluster' => false,
                'default' => [
                    'host'     => '127.0.0.1',
                    'port'     => 6379,
                    'database' => 0,
                ],
            ]
        ],
        ...
    ]

Memcached

    'cache' => [
        'driver'  => 'memcached',
        'options' => [
            'servers' => [
                [
                    'host'   => '127.0.0.1',
                    'port'   => 11211,
                    'weight' => 100,
                ],
            ]
        ],
        ...
    ]

Pass the config repository in the constructor of RequestLimitRuleset

$rules = new RequestLimitRuleset(
    [ ... ], 
    'cache', // caching strategy
    'laravel', // storage adapter
    new Repository(require '../config/laravel-guzzle-limiter.php') // config repository
    );

The same adapter will be used to store the internal request timers.

The adapters can be defined in the ruleset

$rules = new RequestLimitRuleset(
    [ ... ], 
    'cache', // caching strategy
    'array' // storage adapter
    );

Without caching - no-cache

Just throttle the requests. The responses are not cached. Exceeding the rate limits results in a 429 - Too Many Requests exception.

$rules = new RequestLimitRuleset(
    [ ... ], 
    'no-cache', // caching strategy
    'array' // storage adapter
    );

With caching (default) - cache

The middleware tries to fall back to a cached value when the rate limits are exceeded before throwing a 429 - Too Many Requests exception.

$rules = new RequestLimitRuleset(
    [ ... ], 
    'cache', // caching strategy
    'array' // storage adapter
    );

With forced caching - force-cache

Always use cached responses when available to spare your rate limits. As long as there is a response in the cache for the current request, it returns the cached response. It will only actually send the request when no response is in the cache. Otherwise, it throws a 429 - Too Many Requests exception.

You might want to disable the caching of empty responses with this option (see General Driver Settings).

$rules = new RequestLimitRuleset(
    [ ... ], 
    'force-cache', // caching strategy
    'array' // storage adapter
    );

Custom caching strategy

The custom caching strategy must implement the CacheStrategy interface. It is advised to use the Cacheable abstraction to implement base functionality. For reference implementations, please check ForceCache and Cache.

To use the new caching strategy, you'll need to pass the fully qualified class name to RequestLimitRuleset.

Usage

$rules = new RequestLimitRuleset([ ... ], 
                                MyCustomCacheStrategy::class, 
                                'array', 
                                new Repository(...));
                                
$throttle = new ThrottleMiddleware($rules);
...                                

Wildcards

If you want to define the same rules for multiple different hosts, you can use wildcards. A possible use case can be subdomains:

$rules = new RequestLimitRuleset([
        'https://www.{subdomain}.mysite.com' => [
            [
                'max_requests'     => 50,
                'request_interval' => 2
            ]
        ]
    ]);

This host matches https://www.en.mysite.com, https://www.de.mysite.com, https://www.fr.mysite.com, etc.

Changes

Please see CHANGELOG for more information on what has changed recently.

Testing

composer test

Contributing

Please see CONTRIBUTING and CODE_OF_CONDUCT for details.

Security

If you discover any security-related issues, please email chroma91@gmail.com instead of using the issue tracker.

Credits

• Timo Prüße
• Andre Biel
• All Contributors

License

The MIT License (MIT). Please see License File for more information.