This PR was squashed before being merged into the 5.2-dev branch.
Discussion
----------
[HttpClient] Added RetryHttpClient
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| Deprecations? | no
| Tickets | /
| License | MIT
| Doc PR | TODO
This PR adds a new HttpClient decorator to automatically retry failed requests.
When calling API, A very small % of requests are expected to timeout due to transient network issues. Some providers like AWS recommends retrying these requests and use a lower connection timeout so that the clients can fail fast and retry.
I used the almost the same configuration as Messenger
```yaml
framework:
http_client:
default_options:
retry_failed:
enabled: true // default false
decider_service: null
backoff_service: null
http_codes: [423, 425, 429, 500, 502, 503, 504, 507, 510]
max_retries: 3
delay: 1000
multiplier: 2
max_delay: 0
response_header: true
scoped_clients:
github:
scope: 'https://api\.github\.com'
retry_failed:
max_delay: 2000
```
Commits
-------
712ac5999d [HttpClient] Added RetryHttpClient
This PR was merged into the 5.2-dev branch.
Discussion
----------
[Amqp] Add amqps support
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes <!-- please update src/**/CHANGELOG.md files -->
| Deprecations? | no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files -->
| Tickets | Fix#37002
| License | MIT
| Doc PR | symfony/symfony-docs#... TODO <!-- required for new features -->
### Cases
_Case 1 (simple):_
I want to set up my app with a locally installed amqp broker supporting SSL. No auth is needed, no vhost, only SSL certificate file located at /etc/ssl/cafile.
'amqps://'
_Case 2 (complex):_
I use a special port, special file locations for SSL. I pass it through php.ini or configuration.
### Questions
1) Actually AmqpTransportFactory use `supports` function to see if 'amqp://' is contained in dsn string and accept cacert argument to use SSL
How are we supposed to cover AMQPS's case in your opinion?
cacert argument is settable through php.ini and code.
### Observations
I think this PR should aim at:
- Having correct defaults ssl values in case of 'amqps://' eg : case 1
- Being able to pass cacert file path and verify argument. eg : case 2
- Test: but test what?
#### EDIT
As discuted with @nicolas-grekas, we should check that cacert is existing in php.ini or is passed in configuration and throw an exception to give a nice feedback about what's wrong. Php amqp lib isn't actually precise about why it fails when you forget it.
### Steps
- [x] Ensure AMQPS can be used out of the box without DSN.
- [x] If not, try to modify vendor directly to fix it.
- [x] Modify vendor to make AMQPS works with DSN.
- [x] Check cacert to throw a better exception.
- [x] Add test related to last point.
- [ ] Pass CI
Commits
-------
99fc3f3b89 [Amqp] Add amqps support
This PR was squashed before being merged into the 5.2-dev branch.
Discussion
----------
[RFC] Introduce a RateLimiter component
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| Deprecations? | no
| Tickets | Refs #37444
| License | MIT
| Doc PR | tbd
Based on the discussions in #37444, I decided to write a general purpose RateLimiter component. This implementation uses the token bucket algorithm, inspired by the [Go's time/rate](3af7569d3a/rate/rate.go) library and the [PHP `bandwidth-throttle/token-bucket` package](https://github.com/bandwidth-throttle/token-bucket) (which is [unmaintained for years](https://github.com/bandwidth-throttle/token-bucket/issues/19)).
### Usage
The component has two main methods:
* `Limiter::reserve(int $tokens, int $maxTime)`, allocates `$tokens` and returns a `Reservation` containing the wait time. Use this method if your process wants to wait before consuming the token.
* `Limiter::consume(int $tokens)`, checks if `$tokens` are available now and discards the reservation if that's not the case. Use this method if you want to skip when there are not enough tokens at this moment.
The component uses the Lock component to make sure it can be used in parallel processes.
Example:
```php
<?php
namespace App\EventListener;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Event\RequestEvent;
use Symfony\Component\RateLimiter\LimiterFactory;
class LimitListener
{
private $limiterFactory;
public function __construct(LimiterFactory $apiLimiterFactory)
{
$this->limiterFactory = $apiLimiterFactory;
}
public function __invoke(RequestEvent $event)
{
$ip = $event->getRequest()->getClientIp();
$limiter = $this->limiterFactory->createLimiter(preg_replace('/[^a-zA-Z0-9]/', '-', $ip));
if (!$limiter->consume()) {
$event->setResponse(new Response('Too many requests', 429));
}
}
}
```
### Usefullness of the component
I think a generic rate limiter is usefull in quite some places:
* Add a login throttling feature in Symfony
* <s>Rate limiting outgoing API calls (e.g. HttpClient), to prevent hitting upstream API limits.</s> See #37471 (and https://blog.heroku.com/rate-throttle-api-client )
* Allowing users to easily implement API rate limits in their own Symfony-based APIs.
### State of the art
There are some rate limiting packages in PHP, but I think there is no precendent for this component:
* [`graham-campbell/throttle`](https://github.com/GrahamCampbell/Laravel-Throttle) is heavily relying on Laravel. It is however very popular, proofing there is a need for such feature
* [`nikolaposa/rate-limit`](https://github.com/nikolaposa/rate-limit) does not implement reservation of tokens and as such less feature complete. Also its architecture combines the rate limiter and storage, making it harder to implement different storages.
### Todo
If it is agreed that this component can bring something to Symfony, it needs some more love:
* [x] Add more tests
* [x] Integrate with the FrameworkBundle
* [x] Add sliding window implementation
* [x] Add integration with the Security component
* <s>Maybe add more storage implementations? I didn't want to duplicate storage functionalities already existing in the Lock and Cache component, thus I for now focused mostly on integrating the Cache adapters. But maybe a special Doctrine adapter makes sense?</s>
Commits
-------
67417a693e [RFC] Introduce a RateLimiter component
This PR was merged into the 5.2-dev branch.
Discussion
----------
Make RetryTillSaveStore implements the SharedLockStoreInterface
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| Deprecations? | no
| Tickets | /
| License | MIT
| Doc PR | /
The `RetryTillSaveStore` decorates a non blocking store (ie. `RedisStore`) to make it blockable.
The issue is: if the decorate store manage SharedLock, the Decorator does not implements this interface. It's currently not possible to get both `blocking` and `sharable` store.
Commits
-------
a0321e66c9 Make RetryTillSaveStore implements the SharedLockStoreInterface
This PR was merged into the 5.2-dev branch.
Discussion
----------
[Form] Add new way of mapping data using callback functions
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| Deprecations? | yes
| Tickets | Fix#37597 (partially)
| License | MIT
| Doc PR | https://github.com/symfony/symfony-docs/pull/14241
Replaces https://github.com/symfony/symfony/pull/37614
## What this solves
Objects and Forms have different mechanisms for structuring data. When you build an object model with a lot of business logic it's valuable to use these mechanisms to better collect the data and the behavior that goes with it. Doing so leads to variant schemas; that is, the object model schema and the form schema don't match up.
You still need to transfer data between the two schemas, and this data transfer becomes a complexity in its own right. If the objects know about the form structure, changes in one tend to ripple to the other.
Currently, the Data Mapper layer separates the objects from the form, transfering data between the two and also isolating them from each other. That's fine, but at present the default data mapper has a limitation: _it's very tied to one property path_ (see [`PropertyPathMapper`](https://github.com/symfony/symfony/blob/5.1/src/Symfony/Component/Form/Extension/Core/DataMapper/PropertyPathMapper.php)).
That said, you'll have to write your own data mapper in the following situations:
* When the property path differs for reading and writing
* When several form fields are mapped to a single method
* When you need to read data based on the model's state
* When the mapping of the model depends on the submitted form data
* ...
Also, when we create a new data mapper, we usually forget about checking the status of the given data and forms. Whether the data is empty or not; throw an exception if the given data is not an object/array and whether the form field is submitted/synchronized/disabled or not. Not doing that could lead to unwanted behavior.
## What this proposes
Create a new way to write and read values to/from an object/array using callback functions. This feature would be tied to each form field and would also mean a new way of mapping data, but a very convenient one, in which it won't be necessary to define a new data mapper and take into account all what it would imply when you only need to map one field in a different manner or perhaps in only one direction (writing or reading the value).
This PR adds two new options for each form type: `getter` and `setter`, allowed to be `null` or `callable`:
```php
$builder->add('name', TextType::class, [
'getter' => function (Person $person, FormInterface $form): string {
return $person->firstName().' '.$person->lastName();
},
'setter' => function (Person &$person, ?string $name, FormInterface $form): void {
$person->rename($name);
},
]);
```
This would give us the same possibilities as data mappers, but within the form field scope, where:
* `$person` is the view data, basically the underlying data to the form.
* `$form` is the current child form that is being mapped.
* `$name` is the submitted data that belongs to that field.
These two callbacks will be executed following the same rules as for property paths before read and write any value (e.i. early return if empty data, skip mapping if the form field is not mapped or it's disabled, etc).
## What this also proposes
I based the implementation on solving this problem first:
> https://github.com/symfony/symfony/pull/37614#issuecomment-662957865
> [...] the property_path option defines the rules on how it's accessed. From there, the actual way it is accessed (direct property access, accessors, reflection, whatever) are hidden from view. All that matters is the property path (which is deduced from the name if not explicitly set). [...]
So splitting the default data mapper `PropertyPathMapper` into two artifacts: "[DataMapper](https://github.com/yceruto/symfony/blob/data_accessor/src/Symfony/Component/Form/Extension/Core/DataMapper/DataMapper.php)" and "[DataAccessor](https://github.com/yceruto/symfony/blob/data_accessor/src/Symfony/Component/Form/DataAccessorInterface.php)" would allow us adding multiple data accessors along the way (the code that varies in this case) without having to reinvent the wheel over and over again (the data mapper code).
You can also think about a new `ReflectionAccessor` for instance? or use this `CallbackAccessor` to map your form partially from an external API? yes, you could do it :)
Here is a view of the proposed changes:
Where "DataMapper" will take care of common checks, iterates the given child forms, manages the form data and all what is needed for mapping a standard form, whereas "DataAccessor" will take care of how to read and write values to/from the underlying object or array.
## BC
The `PropertyPathMapper` is being deprecated in favor of `DataMapper` class, which uses the `PropertyPathAccessor` by default.
Although `DataMapper` is now the default for each compound form, the behavior must remains the same (tests prove it). So that if `getter` or `setter` option is null (they're by default) the `CallbackAccessor` will falls back to `PropertyPathAccessor` either for reading or writing values.
---
Sorry for the long description, but I think that sometimes it is necessary for too complex issues and big changes. Besides, now you know a little more about what these changes is about.
/cc @xabbuh as creator of [rich-model-forms-bundle](https://github.com/sensiolabs-de/rich-model-forms-bundle) and @alcaeus as you had worked on this issue.
WDYT?
Commits
-------
878effaf47 add new way of mapping data using callback functions
This PR was merged into the 5.1 branch.
Discussion
----------
[Notifier] Fix errors parsing in FirebaseTransport
| Q | A
| ------------- | ---
| Branch? | 5.1
| Bug fix? | yes
| New feature? | no
| Deprecations? | no
| Tickets | /
| License | MIT
| Doc PR | /
I noticed that, when parsing the response, this transport use `getHeader()` like a `string[]` instead of `string[][]`.
I did not test it and I'm not sure about this fix: could you have a look @Jeroeny ?
Commits
-------
2818e2cd43 Fix errors parsing in FirebaseTransport
* 5.1:
Internal classes are not legacy.
[HttpFoundation] Skip the cookie_max_age fixture on PHP 8.
add choice_translation_domain tests to prevent further regressions
Update validators.tr.xlf
* 4.4:
Internal classes are not legacy.
[HttpFoundation] Skip the cookie_max_age fixture on PHP 8.
add choice_translation_domain tests to prevent further regressions
Update validators.tr.xlf
* 3.4:
[HttpFoundation] Skip the cookie_max_age fixture on PHP 8.
add choice_translation_domain tests to prevent further regressions
Update validators.tr.xlf
This PR was merged into the 5.2-dev branch.
Discussion
----------
[Messenger] - Add option to confirm message delivery in Amqp connection
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| Deprecations? | no
| Tickets | -
| License | MIT
| Doc PR | -
Hello! My first PR here.
Sometimes we need to be sure that messages are delivered to Amqp. To make it work we need to wait for the confirmation from the Amqp server. So let's wait for it confirmation if confirmation timeout is defined. Default behaviour are not modified - waiting for confirmation will impact performance of sending message.
Commits
-------
5682d76b2a Messenger - Add option to confirm message delivery in Amqp connection
This PR was merged into the 5.2-dev branch.
Discussion
----------
[Cache] add integration with Messenger to allow computing cached values in a worker
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| BC breaks? | no
| Deprecations? | no
| Tests pass? | yes
| Fixed tickets | -
| License | MIT
| Doc PR | -
~~This PR needs and for now embeds #30334. See 2nd commit.~~
Using the new `CacheInterface` enables probabilistic early expiration by default. This means that from time to time, items are elected as early-expired while they are still fresh ([see Wikipedia](https://en.wikipedia.org/wiki/Cache_stampede#Probabilistic_early_expiration) for details).
This PR adds a new `early_expiration_message_bus` option when configuring cache pools. When this option is set, cache pools are configured to send those early-expired items on a Messenger bus, then serve the current value immediately, while the updated value is computed in a worker in the background.
`CacheInterface::get($key, $callback)` accepts any callable, but sending any callable on a bus is not possible (e.g. a `Closure` cannot be serialized). To bypass this constraint, this feature works only with callables in the form `[$service, 'publicMethod']`, where `$service` is any public or [reversible service](https://github.com/symfony/symfony/pull/30334).
This constraint is a serious one: this $service must compute a value when knowing only its key. This means keys should embed enough information for this to happen. I think that's not that hard - and we may find ways to provide additional context in the future.
At least the goal is worth it: in theory, this strategy allows achieving a 100% hit ratio even when invalidation-by-expiration is used.
There are two things one needs to do to enable this behavior:
1. bind a message bus to a cache pool:
```yaml
framework:
cache:
pools:
test.cache:
early_expiration_message_bus: messenger.default_bus
```
2. route EarlyExpirationMessage to a transport:
```yaml
framework:
messenger:
routing:
'Symfony\Component\Cache\Messenger\EarlyExpirationMessage': amqp
```
Commits
-------
6c0911f58c [Cache] add integration with Messenger to allow computing cached values in a worker
This PR was squashed before being merged into the 5.2-dev branch.
Discussion
----------
[SecurityBundle] Comma separated ips for security.access_control
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | no
| New feature? | yes
| Deprecations? | no
| Tickets |
| License | MIT
| Doc PR | https://github.com/symfony/symfony-docs/pull/14219
There is currently no way to use env vars to configure `security.access_control` ips with multiple values. Ability to use comma separated ips make it able.
Commits
-------
0412e91060 [SecurityBundle] Comma separated ips for security.access_control
* 5.1:
fix merge
[Cache] fix previous PR
add mising sr (latn & cyrl) translations
[Cache] fix ProxyAdapter not persisting items with infinite expiration
[HttpClient] fail properly when the server replies with HTTP/0.9
Fix CS
[Cache] Limit cache version character range
[Mailer] Fixed Mailgun API bridge JsonException when API response is not applicaton/json
[String] improve fix
fix tests
[DI] dump OS-indepent paths in the compiled container
[DI] dump OS-indepent paths in the preload file
Run postgres setup trigger in transaction
allow consumers to mock all methods
* 4.4:
add mising sr (latn & cyrl) translations
[Cache] fix ProxyAdapter not persisting items with infinite expiration
[HttpClient] fail properly when the server replies with HTTP/0.9
Fix CS
[Cache] Limit cache version character range
[DI] dump OS-indepent paths in the compiled container
allow consumers to mock all methods
This PR was merged into the 4.4 branch.
Discussion
----------
[HttpClient] fail properly when the server replies with HTTP/0.9
| Q | A
| ------------- | ---
| Branch? | 4.4
| Bug fix? | no
| New feature? | no
| Deprecations? | no
| Tickets | Fix#37923
| License | MIT
| Doc PR | -
Commits
-------
96759af1da [HttpClient] fail properly when the server replies with HTTP/0.9
This PR was merged into the 5.2-dev branch.
Discussion
----------
Move form error normalizer to the Serializer component
| Q | A
| ------------- | ---
| Branch? | master
| Bug fix? | yes
| New feature? | no <!-- please update src/**/CHANGELOG.md files -->
| Deprecations? | no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files -->
| Tickets | n/a<!-- prefix each issue number with "Fix #", no need to create an issue if none exist, explain below instead -->
| License | MIT
| Doc PR | n/a
Commits
-------
c053db5553 Move form error normalizer to the Serializer component
