merged branch eriksencosta/icu-data-update (PR #5107)

Commits
-------

33105e0 [Locale] added more information to the README.md regarding the need to build the ICU data when the intl extension ICU data version mismatch the one shipped with Symfony
3191c70 [Validator] fixed tests, ICU 4.4 (Travis version) does not have the "my" locale
d7b6bb3 [Locale] updated README.md
6456361 [Locale] added note about ICU data building: use the same PHP intl/ICU version as the desired version to build
025f972 [Locale] renamed function
8da99ca [Locale] updated CHANGELOG.md
d909360 [Locale] don't create a "current" directory anymore, uses only the ICU version as the name of the data directory
3f2b4bf [Locale] changed build data script name
61e3539 [Form][Locale] updated minimum ICU version to 4.0
0d442c7 [Locale] as of ICU 4.8+, slash ("/") and dash ("-") are interchangeable, you can use any of them in a date/time pattern
90d6dc3 [Locale] fixed tests
86da1b3 [Locale] added ICU update-data script to be run before the test suite
5fd4eb1 [Locale] updated ICU data paths
64ccee7 [Locale] updated ICU data
0d0a8d4 [Locale] updated script to generate separated data directories for different ICU releases

Discussion
----------

[2.1][Locale] ICU data update

Bug fix: no
Feature addition: yes
Backwards compatibility break: no
Symfony2 tests pass: no
Fixes the following tickets:
Todo: -
License of the code: MIT

[![Build Status](https://secure.travis-ci.org/eriksencosta/symfony.png?branch=icu-data-update)](http://travis-ci.org/eriksencosta/symfony)

The tests are passing in Travis only because it uses ICU 4.4. The tests fails in ICU 4.8 and 49.1. The data files now are up-to-date with ICU 49.1.2. The fails are consistent between 32 and 64 bit for the PHP 5.3.3, 5.3.14 and 5.4.4 (each one compiled with ICU 4.2 to 49).

Basically tests that relies on Locale will have some idiosyncrasies:

- Some tests will assert a different value for different ICU versions, mostly for currency/date formatting
- Some tests will skip if not the latest ICU release, code that format/parse something that is directly based on ICU behavior will always implement the  behavior from the latest release
- Some tests will make different assertions for different PHP versions (we already have them, just to note)

The Composer script handler was a very quick implementation, I accept suggestions about it. It basically invokes the `update-data.php` script. I will also rename this script later since it now behave a little bit different from the original script.

@fabpot, @stof: should I add mine Composer `post-install-cmd` script in both the project root and component `composer.json` file?

I'm planning to configure a CI server with a setup similar with mine Symfony development environment to run jobs with the different PHP/ICU releases for the Locale component. But most importantly is to document how I did it since it took a lot of time to have an enviroment that supports all this "ICU hell development". Also, there are some pitfalls regarding testing and to generating the resource bundles for the different ICU versions.

Almost there! I'll try to update later tomorrow (monday).

---------------------------------------------------------------------------

by fabpot at 2012-07-30T08:14:29Z

I would not have added the script in the `composer.json` file but in the `.travis.yml` file directly instead. We don't want this script to be run whenever someone run composer on the repository.

---------------------------------------------------------------------------

by eriksencosta at 2012-07-31T03:07:58Z

Ok, moved the script call to the `.travis.yml` file.

---------------------------------------------------------------------------

by eriksencosta at 2012-07-31T12:38:23Z

@stof The last commit was just to backup WIP changes and to ask something to @igorw, it is mine "stage" :)

---------------------------------------------------------------------------

by travisbot at 2012-08-06T03:05:16Z

This pull request [fails](http://travis-ci.org/symfony/symfony/builds/2044244) (merged c6a1dcd1 into 20d2e5a1).

---------------------------------------------------------------------------

by travisbot at 2012-08-06T03:16:12Z

This pull request [fails](http://travis-ci.org/symfony/symfony/builds/2044311) (merged 76868419 into 20d2e5a1).

---------------------------------------------------------------------------

by travisbot at 2012-08-06T03:52:20Z

This pull request [fails](http://travis-ci.org/symfony/symfony/builds/2044425) (merged 7e768eb2 into 20d2e5a1).

---------------------------------------------------------------------------

by travisbot at 2012-08-06T04:04:30Z

This pull request [fails](http://travis-ci.org/symfony/symfony/builds/2044461) (merged edaca423 into 20d2e5a1).

---------------------------------------------------------------------------

by eriksencosta at 2012-08-06T04:33:55Z

Ready for primetime. The failed test is not related with mine changes.

---------------------------------------------------------------------------

by travisbot at 2012-08-07T02:32:19Z

This pull request [passes](http://travis-ci.org/symfony/symfony/builds/2053806) (merged 386d20c8 into 842b599c).

---------------------------------------------------------------------------

by fabpot at 2012-08-07T13:51:59Z

After merging this PR, I have more failing tests on my machine:

PHP 5.3.8 - ICU 4.4.1

Is it expected? Is yes, how do we need to run the tests locally?

---------------------------------------------------------------------------

by eriksencosta at 2012-08-07T14:56:23Z

@fabpot Yep, it is expected. You'll need to build the ICU data for this version and export the `USE_INTL_ICU_DATA_VERSION` before running the tests:

    php src/Symfony/Component/Locale/Resources/data/build-data.php
    export USE_INTL_ICU_DATA_VERSION=1
    phpunit src/Symfony/Component/Locale/Tests

The errors happens because the implementation is in sync with the behavior of ICU 49.1.2 and the shipped data is from data version too. Maybe I should add the steps above in the components `README.md` since it is more straightforward.

---------------------------------------------------------------------------

by fabpot at 2012-08-07T14:58:44Z

@eriksencosta Yes, that would be good to add some more information in the README file.

---------------------------------------------------------------------------

by eriksencosta at 2012-08-07T15:21:04Z

@fabpot Done!

---------------------------------------------------------------------------

by eriksencosta at 2012-08-07T15:21:29Z

And rebased.

.travis.yml

@@ -8,3 +8,5 @@ php:
 before_script:
     - curl -s http://getcomposer.org/installer | php
     - COMPOSER_ROOT_VERSION=dev-master php composer.phar --dev install
+    - php src/Symfony/Component/Locale/Resources/data/build-data.php
+    - export USE_INTL_ICU_DATA_VERSION=1

src/Symfony/Component/Form/Tests/Extension/Core/DataTransformer/DateTimeToLocalizedStringTransformerTest.php

@@ -89,18 +89,22 @@ class DateTimeToLocalizedStringTransformerTest extends DateTimeTestCase
     {
         $transformer = new DateTimeToLocalizedStringTransformer('UTC', 'UTC', null, \IntlDateFormatter::LONG);
 
-        $this->assertEquals('03.02.2010 04:05:06 GMT+00:00', $transformer->transform($this->dateTime));
+        $expected = $this->isLowerThanIcuVersion('4.8') ? '03.02.2010 04:05:06 GMT+00:00' : '03.02.2010 04:05:06 GMT';
+
+        $this->assertEquals($expected, $transformer->transform($this->dateTime));
     }
 
     public function testTransformFullTime()
     {
-        if ($this->isLowerThanIcuVersion(3.8)) {
-            $this->markTestSkipped('Please upgrade ICU version to 3.8+');
+        if ($this->isLowerThanIcuVersion('4.0')) {
+            $this->markTestSkipped('Please upgrade ICU version to 4.0+');
         }
 
         $transformer = new DateTimeToLocalizedStringTransformer('UTC', 'UTC', null, \IntlDateFormatter::FULL);
 
-        $this->assertEquals('03.02.2010 04:05:06 GMT+00:00', $transformer->transform($this->dateTime));
+        $expected = $this->isLowerThanIcuVersion('4.8') ? '03.02.2010 04:05:06 GMT+00:00' : '03.02.2010 04:05:06 GMT';
+
+        $this->assertEquals($expected, $transformer->transform($this->dateTime));
     }
 
     public function testTransformToDifferentLocale()
@@ -209,8 +213,8 @@ class DateTimeToLocalizedStringTransformerTest extends DateTimeTestCase
 
     public function testReverseTransformFullTime()
     {
-        if ($this->isLowerThanIcuVersion(3.8)) {
-            $this->markTestSkipped('Please upgrade ICU version to 3.8+');
+        if ($this->isLowerThanIcuVersion('4.0')) {
+            $this->markTestSkipped('Please upgrade ICU version to 4.0+');
         }
 
         $transformer = new DateTimeToLocalizedStringTransformer('UTC', 'UTC', null, \IntlDateFormatter::FULL);

src/Symfony/Component/Locale/CHANGELOG.md

@@ -4,4 +4,6 @@ CHANGELOG
 2.1.0
 -----
 
- * added Locale::getIcuVersion() and Locale::getIcuDataVersion()
+ * added Locale::getIntlIcuVersion(), Locale::getIntlIcuDataVersion(), Locale::getIcuDataVersion() and Locale::getIcuDataDirectory()
+ * renamed update-data.php to build-data.php, the script usage changed, now it is easier to update the ICU data
+ * updated the ICU data to the release 49.1.2

src/Symfony/Component/Locale/Locale.php

@@ -13,6 +13,11 @@ namespace Symfony\Component\Locale;
 
 class Locale extends \Locale
 {
+    /**
+     * The ICU data version that ships with Symfony
+     */
+    const ICU_DATA_VERSION = '49';
+
     /**
      * Caches the countries in different locales
      * @var array
@@ -43,7 +48,7 @@ class Locale extends \Locale
     public static function getDisplayCountries($locale)
     {
         if (!isset(self::$countries[$locale])) {
-            $bundle = \ResourceBundle::create($locale, __DIR__.'/Resources/data/region');
+            $bundle = \ResourceBundle::create($locale, self::getIcuDataDirectory().'/region');
 
             if (null === $bundle) {
                 throw new \RuntimeException(sprintf('The country resource bundle could not be loaded for locale "%s"', $locale));
@@ -98,7 +103,7 @@ class Locale extends \Locale
     public static function getDisplayLanguages($locale)
     {
         if (!isset(self::$languages[$locale])) {
-            $bundle = \ResourceBundle::create($locale, __DIR__.'/Resources/data/lang');
+            $bundle = \ResourceBundle::create($locale, self::getIcuDataDirectory().'/lang');
 
             if (null === $bundle) {
                 throw new \RuntimeException(sprintf('The language resource bundle could not be loaded for locale "%s"', $locale));
@@ -149,7 +154,7 @@ class Locale extends \Locale
     public static function getDisplayLocales($locale)
     {
         if (!isset(self::$locales[$locale])) {
-            $bundle = \ResourceBundle::create($locale, __DIR__.'/Resources/data/names');
+            $bundle = \ResourceBundle::create($locale, self::getIcuDataDirectory().'/names');
 
             if (null === $bundle) {
                 throw new \RuntimeException(sprintf('The locale resource bundle could not be loaded for locale "%s"', $locale));
@@ -188,11 +193,11 @@ class Locale extends \Locale
     }
 
     /**
-     * Returns the ICU version
+     * Returns the ICU version as defined by the intl extension
      *
      * @return string|null The ICU version
      */
-    public static function getIcuVersion()
+    public static function getIntlIcuVersion()
     {
         if (defined('INTL_ICU_VERSION')) {
             return INTL_ICU_VERSION;
@@ -213,11 +218,11 @@ class Locale extends \Locale
     }
 
     /**
-     * Returns the ICU Data version
+     * Returns the ICU Data version as defined by the intl extension
      *
      * @return string|null The ICU Data version
      */
-    public static function getIcuDataVersion()
+    public static function getIntlIcuDataVersion()
     {
         if (defined('INTL_ICU_DATA_VERSION')) {
             return INTL_ICU_DATA_VERSION;
@@ -237,6 +242,41 @@ class Locale extends \Locale
         return trim($matches[1]);
     }
 
+    /**
+     * Returns the ICU data version that ships with Symfony. If the environment variable USE_INTL_ICU_DATA_VERSION is
+     * defined, it will try use the ICU data version as defined by the intl extension, if available.
+     *
+     * @return string The ICU data version that ships with Symfony
+     */
+    public static function getIcuDataVersion()
+    {
+        static $dataVersion;
+
+        if (null === $dataVersion) {
+            $dataVersion = self::ICU_DATA_VERSION;
+
+            if (getenv('USE_INTL_ICU_DATA_VERSION') && self::getIntlIcuVersion()) {
+                $dataVersion = self::getIntlIcuVersion();
+
+                preg_match('/^(?P<version>[0-9]\.[0-9]|[0-9]{2,})/', $dataVersion, $matches);
+
+                $dataVersion = $matches['version'];
+            }
+        }
+
+        return $dataVersion;
+    }
+
+    /**
+     * Returns the directory path of the ICU data that ships with Symfony
+     *
+     * @return string The path to the ICU data directory
+     */
+    public static function getIcuDataDirectory()
+    {
+        return __DIR__.'/Resources/data/'.self::getIcuDataVersion();
+    }
+
     /**
      * Returns the fallback locale for a given locale, if any
      *

src/Symfony/Component/Locale/README.md

@@ -20,3 +20,17 @@ Resources
 You can run the unit tests with the following command:
 
     phpunit
+
+If your PHP have the ``intl`` extension enabled but the intl extension ICU data
+version mismatch the one shipped with the component, you can build the data for
+it and use the ``USE_INTL_ICU_DATA_VERSION`` environment variable.
+
+    php Resources/data/build-data.php
+    export USE_INTL_ICU_DATA_VERSION=true
+    phpunit
+
+This way the tests will use the ICU data files with the same version of your
+``intl`` extension.
+
+Read the file ``Resources/data/UPDATE.txt`` for more info about building or
+updating the ICU data files.