diogo/symfony
Archived
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

merged branch stof/gettext_loader (PR #2412)

Browse Source 
Commits
-------

d974a4a Merge pull request #4 from stealth35/test_mo_loader
cf05646 delete useless tests
19f9de9 [Translation] fix gettext tests
965f2bf Merge pull request #3 from stealth35/test_mo_loader
9c2a26d [Translation] add Mo loader tests
9af2342 [Translation] Added the gettext loaders

Discussion
----------

[Translation] Added the gettext loaders

This is the squashed version of the work done by @xaav in #634.

@stealth35 you said you will work on the dumpers. do you have some stuff on it ?

---------------------------------------------------------------------------

by drak at 2011/10/24 19:28:43 -0700

Is there any more progress with this?

---------------------------------------------------------------------------

by stealth35 at 2011/10/25 00:57:19 -0700

I work on the dumpers, but the Po loader is wrong, caus' the Po ressource can be multiline,

     msgid ""
     "Here is an example of how one might continue a very long string\n"
     "for the common case the string represents multi-line output.\n"

http://www.gnu.org/software/gettext/manual/gettext.html#PO-Files

Anyway the Po format is an intermediate format to Mo file, (like .txt to .res file for ICU), IMO we can just support the real gettext format : Mo

---------------------------------------------------------------------------

by stealth35 at 2011/11/03 02:00:24 -0700

@stof The MO Dumper is ready (stealth35/symfony@f2d1d5b4de), should we keep the PO format ?

---------------------------------------------------------------------------

by fabpot at 2011/11/07 08:50:59 -0800

@stealth35: The PO is what people will use for their translations. They will then dump it to MO. So, we need both PO and MO loaders and dumpers.

---------------------------------------------------------------------------

by stealth35 at 2011/11/08 01:25:39 -0800

@fabpot, I'm ready for both dumpers, you can merge this, and I'll open a PR for the dumpers

---------------------------------------------------------------------------

by fabpot at 2011/11/08 22:37:47 -0800

I've just had a look at this PR code again and I see that the unit tests are pretty slim. Is it possible to add some tests for the mo loader?

---------------------------------------------------------------------------

by stealth35 at 2011/11/09 01:15:25 -0800

@fabpot test send to @stof ✌️

---------------------------------------------------------------------------

by stof at 2011/11/09 02:22:55 -0800

and merged in this branch

---------------------------------------------------------------------------

by fabpot at 2011/11/09 02:39:09 -0800

The tests do not pass for me:

    There was 1 error:

    1) Symfony\Tests\Component\Translation\Loader\MoFileLoaderTest::testLoadDoesNothingIfEmpty
    InvalidArgumentException: MO stream content has an invalid format.

    /Users/fabien/work/symfony/git/symfony/src/Symfony/Component/Translation/Loader/MoFileLoader.php:79
    /Users/fabien/work/symfony/git/symfony/src/Symfony/Component/Translation/Loader/MoFileLoader.php:46
    /Users/fabien/work/symfony/git/symfony/tests/Symfony/Tests/Component/Translation/Loader/MoFileLoaderTest.php:34

    --

    There was 1 failure:

    1) Symfony\Tests\Component\Translation\Loader\PoFileLoaderTest::testLoad
    Failed asserting that two arrays are equal.
    --- Expected
    +++ Actual
    @@ @@
     Array (
    -    'foo' => 'bar'
     )

    /Users/fabien/work/symfony/git/symfony/tests/Symfony/Tests/Component/Translation/Loader/PoFileLoaderTest.php:25
This commit is contained in:
Fabien Potencier 2011-11-09 22:00:02 +01:00
commit 8ca8aef151
9 changed files with 365 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
src/Symfony/Bundle/FrameworkBundle/Resources/config/translation.xml
View File

@ -11,6 +11,8 @@
<parameter key="translation.loader.php.class">Symfony\Component\Translation\Loader\PhpFileLoader</parameter>
<parameter key="translation.loader.yml.class">Symfony\Component\Translation\Loader\YamlFileLoader</parameter>
<parameter key="translation.loader.xliff.class">Symfony\Component\Translation\Loader\XliffFileLoader</parameter>
<parameter key="translation.loader.po.class">Symfony\Component\Translation\Loader\PoFileLoader</parameter>
<parameter key="translation.loader.mo.class">Symfony\Component\Translation\Loader\MoFileLoader</parameter>
<parameter key="translation.loader.qt.class">Symfony\Component\Translation\Loader\QtTranslationsLoader</parameter>
<parameter key="translation.loader.csv.class">Symfony\Component\Translation\Loader\CsvFileLoader</parameter>
<parameter key="translation.loader.rb.class">Symfony\Component\Translation\Loader\ResourceBundleLoader</parameter>
@ -56,6 +58,14 @@
<tag name="translation.loader" alias="xliff" />
</service>
<service id="translation.loader.po" class="%translation.loader.po.class%">
<tag name="translation.loader" alias="po" />
</service>
<service id="translation.loader.mo" class="%translation.loader.mo.class%">
<tag name="translation.loader" alias="mo" />
</service>
<service id="translation.loader.qt" class="%translation.loader.qt.class%">
<tag name="translation.loader" alias="ts" />
</service>

169
src/Symfony/Component/Translation/Loader/MoFileLoader.php Normal file
View File

@ -0,0 +1,169 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Component\Translation\Loader;
use Symfony\Component\Config\Resource\FileResource;
/**
* @copyright Copyright (c) 2010, Union of RAD http://union-of-rad.org (http://lithify.me/)
*/
class MoFileLoader extends ArrayLoader implements LoaderInterface
{
/**
* Magic used for validating the format of a MO file as well as
* detecting if the machine used to create that file was little endian.
*
* @var float
*/
const MO_LITTLE_ENDIAN_MAGIC = 0x950412de;
/**
* Magic used for validating the format of a MO file as well as
* detecting if the machine used to create that file was big endian.
*
* @var float
*/
const MO_BIG_ENDIAN_MAGIC = 0xde120495;
/**
* The size of the header of a MO file in bytes.
*
* @var integer Number of bytes.
*/
const MO_HEADER_SIZE = 28;
public function load($resource, $locale, $domain = 'messages')
{
$messages = $this->parse($resource);
// empty file
if (null === $messages) {
$messages = array();
}
// not an array
if (!is_array($messages)) {
throw new \InvalidArgumentException(sprintf('The file "%s" must contain a valid mo file.', $resource));
}
$catalogue = parent::load($messages, $locale, $domain);
$catalogue->addResource(new FileResource($resource));
return $catalogue;
}
/**
* Parses machine object (MO) format, independent of the machine's endian it
* was created on. Both 32bit and 64bit systems are supported.
*
* @param resource $stream
* @return array
* @throws InvalidArgumentException If stream content has an invalid format.
*/
private function parse($resource)
{
$stream = fopen($resource, 'r');
$stat = fstat($stream);
if ($stat['size'] < self::MO_HEADER_SIZE) {
throw new \InvalidArgumentException("MO stream content has an invalid format.");
}
$magic = unpack('V1', fread($stream, 4));
$magic = hexdec(substr(dechex(current($magic)), -8));
if ($magic == self::MO_LITTLE_ENDIAN_MAGIC) {
$isBigEndian = false;
} elseif ($magic == self::MO_BIG_ENDIAN_MAGIC) {
$isBigEndian = true;
} else {
throw new \InvalidArgumentException("MO stream content has an invalid format.");
}
$header = array(
'formatRevision' => null,
'count' => null,
'offsetId' => null,
'offsetTranslated' => null,
'sizeHashes' => null,
'offsetHashes' => null,
);
foreach ($header as &$value) {
$value = $this->readLong($stream, $isBigEndian);
}
extract($header);
$messages = array();
for ($i = 0; $i < $count; $i++) {
$singularId = $pluralId = null;
$translated = null;
fseek($stream, $offsetId + $i * 8);
$length = $this->readLong($stream, $isBigEndian);
$offset = $this->readLong($stream, $isBigEndian);
if ($length < 1) {
continue;
}
fseek($stream, $offset);
$singularId = fread($stream, $length);
if (strpos($singularId, "\000") !== false) {
list($singularId, $pluralId) = explode("\000", $singularId);
}
fseek($stream, $offsetTranslated + $i * 8);
$length = $this->readLong($stream, $isBigEndian);
$offset = $this->readLong($stream, $isBigEndian);
fseek($stream, $offset);
$translated = fread($stream, $length);
if (strpos($translated, "\000") !== false) {
$translated = explode("\000", $translated);
}
$ids = array('singular' => $singularId, 'plural' => $pluralId);
$item = compact('ids', 'translated');
if (is_array($item['translated'])) {
$messages[$item['ids']['singular']] = stripslashes($item['translated'][0]);
if (isset($item['ids']['plural'])) {
$messages[$item['ids']['plural']] = stripslashes(end($item['translated']));
}
} elseif($item['ids']['singular']) {
$messages[$item['ids']['singular']] = stripslashes($item['translated']);
}
}
fclose($stream);
return array_filter($messages);
}
/**
* Reads an unsigned long from stream respecting endianess.
*
* @param resource $stream
* @param boolean $isBigEndian
* @return integer
*/
private function readLong($stream, $isBigEndian)
{
$result = unpack($isBigEndian ? 'N1' : 'V1', fread($stream, 4));
$result = current($result);
return (integer) substr($result, -8);
}
}

104
src/Symfony/Component/Translation/Loader/PoFileLoader.php Normal file
View File

@ -0,0 +1,104 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Component\Translation\Loader;
use Symfony\Component\Config\Resource\FileResource;
/**
* @copyright Copyright (c) 2010, Union of RAD http://union-of-rad.org (http://lithify.me/)
*/
class PoFileLoader extends ArrayLoader implements LoaderInterface
{
public function load($resource, $locale, $domain = 'messages')
{
$messages = $this->parse($resource);
// empty file
if (null === $messages) {
$messages = array();
}
// not an array
if (!is_array($messages)) {
throw new \InvalidArgumentException(sprintf('The file "%s" must contain a valid po file.', $resource));
}
$catalogue = parent::load($messages, $locale, $domain);
$catalogue->addResource(new FileResource($resource));
return $catalogue;
}
/**
* Parses portable object (PO) format.
*
* This parser sacrifices some features of the reference implementation the
* differences to that implementation are as follows.
* - No support for comments spanning multiple lines.
* - Translator and extracted comments are treated as being the same type.
* - Message IDs are allowed to have other encodings as just US-ASCII.
*
* Items with an empty id are ignored.
*
* @param resource $stream
* @return array
*/
private function parse($resource)
{
$stream = fopen($resource, 'r');
$defaults = array(
'ids' => array(),
'translated' => null,
);
$messages = array();
$item = $defaults;
while ($line = fgets($stream)) {
$line = trim($line);
if ($line === '') {
if (is_array($item['translated'])) {
$messages[$item['ids']['singular']] = stripslashes($item['translated'][0]);
if (isset($item['ids']['plural'])) {
$messages[$item['ids']['plural']] = stripslashes(end($item['translated']));
}
} elseif($item['ids']['singular']) {
$messages[$item['ids']['singular']] = stripslashes($item['translated']);
}
$item = $defaults;
} elseif (substr($line, 0, 7) === 'msgid "') {
$item['ids']['singular'] = substr($line, 7, -1);
} elseif (substr($line, 0, 8) === 'msgstr "') {
$item['translated'] = substr($line, 8, -1);
} elseif ($line[0] === '"') {
$continues = isset($item['translated']) ? 'translated' : 'ids';
if (is_array($item[$continues])) {
end($item[$continues]);
$item[$continues][key($item[$continues])] .= substr($line, 1, -1);
} else {
$item[$continues] .= substr($line, 1, -1);
}
} elseif (substr($line, 0, 14) === 'msgid_plural "') {
$item['ids']['plural'] = substr($line, 14, -1);
} elseif (substr($line, 0, 7) === 'msgstr[') {
$item['translated'][(integer) substr($line, 7, 1)] = substr($line, 11, -1);
}
}
fclose($stream);
return array_filter($messages);
}
}

39
tests/Symfony/Tests/Component/Translation/Loader/MoFileLoaderTest.php Normal file
View File

@ -0,0 +1,39 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Tests\Component\Translation\Loader;
use Symfony\Component\Translation\Loader\MoFileLoader;
use Symfony\Component\Config\Resource\FileResource;
class MoFileLoaderTest extends \PHPUnit_Framework_TestCase
{
public function testLoad()
{
$loader = new MoFileLoader();
$resource = __DIR__.'/../fixtures/resources.mo';
$catalogue = $loader->load($resource, 'en', 'domain1');
$this->assertEquals(array('foo' => 'bar'), $catalogue->all('domain1'));
$this->assertEquals('en', $catalogue->getLocale());
$this->assertEquals(array(new FileResource($resource)), $catalogue->getResources());
}
/**
* @expectedException \InvalidArgumentException
*/
public function testLoadInvalidResource()
{
$loader = new MoFileLoader();
$resource = __DIR__.'/../fixtures/empty.mo';
$catalogue = $loader->load($resource, 'en', 'domain1');
}
}

40
tests/Symfony/Tests/Component/Translation/Loader/PoFileLoaderTest.php Normal file
View File

@ -0,0 +1,40 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Tests\Component\Translation\Loader;
use Symfony\Component\Translation\Loader\PoFileLoader;
use Symfony\Component\Config\Resource\FileResource;
class PoFileLoaderTest extends \PHPUnit_Framework_TestCase
{
public function testLoad()
{
$loader = new PoFileLoader();
$resource = __DIR__.'/../fixtures/resources.po';
$catalogue = $loader->load($resource, 'en', 'domain1');
$this->assertEquals(array('foo' => 'bar'), $catalogue->all('domain1'));
$this->assertEquals('en', $catalogue->getLocale());
$this->assertEquals(array(new FileResource($resource)), $catalogue->getResources());
}
public function testLoadDoesNothingIfEmpty()
{
$loader = new PoFileLoader();
$resource = __DIR__.'/../fixtures/empty.po';
$catalogue = $loader->load($resource, 'en', 'domain1');
$this->assertEquals(array(), $catalogue->all('domain1'));
$this->assertEquals('en', $catalogue->getLocale());
$this->assertEquals(array(new FileResource($resource)), $catalogue->getResources());
}
}

0
tests/Symfony/Tests/Component/Translation/fixtures/empty.mo Normal file
View File

0
tests/Symfony/Tests/Component/Translation/fixtures/empty.po Normal file
View File

BIN
tests/Symfony/Tests/Component/Translation/fixtures/resources.mo Normal file
View File

Binary file not shown.

3
tests/Symfony/Tests/Component/Translation/fixtures/resources.po Normal file
View File

@ -0,0 +1,3 @@
msgid "foo"
msgstr "bar"