diogo/symfony
Archived
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

merged branch alexandresalome/new-profiler-storage (PR #1772)

Browse Source 
Commits
-------

9f0bd03 [HttpKernel] Update tests for FileProfilerStorage
b7032bc [HttpKernel] Update FileProfileStorage to search from EOF
188a5fa [HttpKernel] Override the existing tokens in FileProfilerStorage
b1b1424 [HttpKernel] Delete folders in the profiler cache
88bc3ec [HttpKernel] Fixes standards of FileProfilerStorage
affe66c Merge remote-tracking branch 'origin/master' into new-profiler-storage
ea916c3 [HttpKernel] Coding convention for the file profiler storage
9ae2c8d [HttpKernel] CS in file storage
b415efd [HttpKernel] Add a test for semicolon in file storage test
1c1215f [HttpKernel] Use subfolders for better storage in file storage of profiler
4b1dc1f [HttpKernel] Fix the folder attribute of file storage to private
70f73e1 [HttpKernel] Fix tests for the file storage of profiler
d5313d9 [HttpKernel] Add tests for the file profiler storage
09fc0a2 [HttpKernel] Add Symfony credits to the file storage class for the profiler
d1d5892 [HttpKernel] Finalize the file storage for the profiler
2f65cf2 Add POC for file storage system

Discussion
----------

[2.1] [HttpKernel] File storage for profiler

Symfony2 has some problems when dealing with multiple concurrency queries in the SQLite storage, resulting in a timeout error or terrible lack.

I've implemented after discussions with @fabpot a filesystem storage.

Enable it in your project with :

    framework:
        profiler:
          dsn:             "file:%kernel.cache_dir%/profiler"

I also studied the possibility to store only big data string in files and rest in the SQLite, but not concluant.

Results of my measures (4 concurrency, 120 total) :

* SQLite with data : 1057ms
* SQLite without data : 615ms
* MySQL : 40ms
* This File storage : 54ms

---------------------------------------------------------------------------

by alexandresalome at 2011/07/22 12:01:10 -0700

An idea for the find method : a csv file containing ip;url;token

The iteration could be done over a big file, without loading the whole file in memory.

---------------------------------------------------------------------------

by alexandresalome at 2011/07/23 14:22:32 -0700

OK new version, with as explained previously : a CSV file containing the index + file for each profile.

The speed is similar to the speed of MySQL, and no memory overhead should occur with this solution.

---------------------------------------------------------------------------

by alexandresalome at 2011/07/23 14:37:14 -0700

Hm... Created tests, duplicated from SqliteProfilerStorageTest.

Any idea on how to put this code in common ? Is it usual to create a base class for 2 tests ?

---------------------------------------------------------------------------

by alexandresalome at 2011/07/23 14:48:39 -0700

Just tested with 24.000 requests, the 24.001'th request still takes less than 50ms to execute.

The index file is about 2Mb, and iterating the whole file is fast.

---------------------------------------------------------------------------

by alexandresalome at 2011/07/23 14:53:19 -0700

I've filled the file with 120Mb of data, requests are still less than 50ms for executing.

Iterating the index takes more than 30s (so it crashed), but it's because of the amount of lines. 30 seconds = 1,400,000 lines in this computer. The file = 1,500,000 lines

---------------------------------------------------------------------------

by alexandresalome at 2011/07/23 14:56:54 -0700

I've tested it with Linux, is someone can test with Windows

---------------------------------------------------------------------------

by stloyd at 2011/07/24 00:32:32 -0700

IMO to speedup it a bit more and not end up with "crash" (to not end with "limit" of files per directory, also to many files in dir slow down every OS) you should use same method to write as Twig, split up files in to directories. If you do this you can speed up index more, because you can create smaller one per directory.

Also you should fix CS (coding standards).

---------------------------------------------------------------------------

by stloyd at 2011/07/24 02:10:20 -0700

Tested on Win 7, seems ok. Similar speed to sqlite, dunno why ;-) but used a bit less of memory.

---------------------------------------------------------------------------

by alexandresalome at 2011/07/24 02:13:21 -0700

Did you tried with concurrent requests ? It makes sense when you use assetic
and your browser hits the application 4 times simultaneously for CSS
generation

---------------------------------------------------------------------------

by alexandresalome at 2011/07/24 02:17:23 -0700

I used Apache Benchmark for producing results :

    ab -c 4 -n 120 URL

---------------------------------------------------------------------------

by alexandresalome at 2011/07/24 02:56:55 -0700

OK I used subfolders, based on last characters (because the first part of token is mostly the same between queries.

---------------------------------------------------------------------------

by stof at 2011/09/04 01:27:15 -0700

@fabpot any news about it ? Can it be merged ?
This commit is contained in:
Fabien Potencier 2011-09-06 07:45:55 +02:00
commit ed472d36b7
3 changed files with 374 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php
View File

@ -196,6 +196,7 @@ class FrameworkExtension extends Extension
$supported = array(
'sqlite' => 'Symfony\Component\HttpKernel\Profiler\SqliteProfilerStorage',
'mysql' => 'Symfony\Component\HttpKernel\Profiler\MysqlProfilerStorage',
'file' => 'Symfony\Component\HttpKernel\Profiler\FileProfilerStorage',
);
list($class, ) = explode(':', $config['dsn']);
if (!isset($supported[$class])) {

223
src/Symfony/Component/HttpKernel/Profiler/FileProfilerStorage.php Normal file
View File

@ -0,0 +1,223 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Component\HttpKernel\Profiler;
/**
* Storage for profiler using files.
*
* @author Alexandre Salomé <alexandre.salome@gmail.com>
*/
class FileProfilerStorage implements ProfilerStorageInterface
{
/**
* Folder where profiler data are stored.
*
* @var string
*/
private $folder;
/**
* Constructs the file storage using a "dsn-like" path.
*
* Example : "file:/path/to/the/storage/folder"
*
* @param string $dsn The DSN
*/
public function __construct($dsn)
{
if (0 !== strpos($dsn, 'file:')) {
throw new \InvalidArgumentException("FileStorage DSN must start with file:");
}
$this->folder = substr($dsn, 5);
if (!is_dir($this->folder)) {
mkdir($this->folder);
}
}
/**
* {@inheritdoc}
*/
public function find($ip, $url, $limit)
{
$file = $this->getIndexFilename();
if (!file_exists($file)) {
return array();
}
$file = fopen($file, 'r');
fseek($file, 0, SEEK_END);
$result = array();
while ($limit > 0) {
$line = $this->readLineFromFile($file);
if (false === $line) {
break;
}
if ($line === "") {
continue;
}
list($csvToken, $csvIp, $csvUrl, $csvTime, $csvParent) = str_getcsv($line);
if ($ip && false === strpos($csvIp, $ip) || $url && false === strpos($csvUrl, $url)) {
continue;
}
$row = array(
'token' => $csvToken,
'ip' => $csvIp,
'url' => $csvUrl,
'time' => $csvTime,
'parent' => $csvParent
);
$result[] = $row;
$limit--;
}
fclose($file);
return $result;
}
/**
* {@inheritdoc}
*/
public function purge()
{
$flags = \FilesystemIterator::SKIP_DOTS;
$iterator = new \RecursiveDirectoryIterator($this->folder, $flags);
$iterator = new \RecursiveIteratorIterator($iterator, \RecursiveIteratorIterator::CHILD_FIRST);
foreach ($iterator as $file) {
if (is_file($file)) {
unlink($file);
} else {
rmdir($file);
}
}
}
/**
* {@inheritdoc}
*/
public function read($token)
{
$file = $this->getFilename($token);
if (!file_exists($file)) {
return null;
}
return unserialize(file_get_contents($file));
}
/**
* {@inheritdoc}
*/
public function write(Profile $profile)
{
$file = $this->getFilename($profile->getToken());
$exists = file_exists($file);
// Create directory
$dir = dirname($file);
if (!is_dir($dir)) {
mkdir($dir, 0777, true);
}
// Store profile
file_put_contents($file, serialize($profile));
// Add to index
$file = fopen($this->getIndexFilename(), 'a');
fputcsv($file, array(
$profile->getToken(),
$profile->getIp(),
$profile->getUrl(),
$profile->getTime(),
$profile->getParent() ? $profile->getParent()->getToken() : null
));
fclose($file);
return ! $exists;
}
/**
* Gets filename to store data, associated to the token.
*
* @return string The profile filename
*/
protected function getFilename($token)
{
// Uses 4 last characters, because first are mostly the same.
$folderA = substr($token, -2, 2);
$folderB = substr($token, -4, 2);
return $this->folder.'/'.$folderA.'/'.$folderB.'/'.$token;
}
/**
* Gets the index filename.
*
* @return string The index filename
*/
protected function getIndexFilename()
{
return $this->folder.'/'.'index.csv';
}
/**
* Reads a line in the file, ending with the current position.
*
* This function automatically skips the empty lines and do not include the line return in result value.
*
* @param resource $file The file resource, with the pointer placed at the end of the line to read
*
* @return mixed A string representating the line or FALSE if beginning of file is reached
*/
protected function readLineFromFile($file)
{
if (ftell($file) === 0) {
return false;
}
fseek($file, -1, SEEK_CUR);
$str = '';
while (true) {
$char = fgetc($file);
if ($char === "\n") {
// Leave the file with cursor before the line return
fseek($file, -1, SEEK_CUR);
break;
}
$str = $char . $str;
if (ftell($file) === 1) {
// All file is read, so we move cursor to the position 0
fseek($file, -1, SEEK_CUR);
break;
}
fseek($file, -2, SEEK_CUR);
}
return $str === "" ? $this->readLineFromFile($file) : $str;
}
}

150
tests/Symfony/Tests/Component/HttpKernel/Profiler/FileProfilerStorageTest.php Normal file
View File

@ -0,0 +1,150 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Tests\Component\HttpKernel\Profiler;
use Symfony\Component\HttpKernel\Profiler\FileProfilerStorage;
use Symfony\Component\HttpKernel\Profiler\Profile;
class FileProfilerStorageTest extends \PHPUnit_Framework_TestCase
{
protected static $tmpDir;
protected static $storage;
protected static function cleanDir()
{
$flags = \FilesystemIterator::SKIP_DOTS;
$iterator = new \RecursiveDirectoryIterator(self::$tmpDir, $flags);
$iterator = new \RecursiveIteratorIterator($iterator, \RecursiveIteratorIterator::SELF_FIRST);
foreach ($iterator as $file)
{
if (is_file($file)) {
unlink($file);
}
}
}
public static function setUpBeforeClass()
{
self::$tmpDir = sys_get_temp_dir() . '/sf2_profiler_file_storage';
if (is_dir(self::$tmpDir)) {
self::cleanDir();
}
self::$storage = new FileProfilerStorage('file:'.self::$tmpDir);
}
public static function tearDownAfterClass()
{
self::cleanDir();
}
protected function setUp()
{
self::$storage->purge();
}
public function testStore()
{
for ($i = 0; $i < 10; $i ++) {
$profile = new Profile('token_'.$i);
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo.bar');
self::$storage->write($profile);
}
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo.bar', 20)), 10, '->write() stores data in the database');
}
public function testStoreSpecialCharsInUrl()
{
// The SQLite storage accepts special characters in URLs (Even though URLs are not
// supposed to contain them)
$profile = new Profile('simple_quote');
$profile->setUrl('127.0.0.1', 'http://foo.bar/\'');
self::$storage->write($profile);
$this->assertTrue(false !== self::$storage->read('simple_quote'), '->write() accepts single quotes in URL');
$profile = new Profile('double_quote');
$profile->setUrl('127.0.0.1', 'http://foo.bar/"');
self::$storage->write($profile);
$this->assertTrue(false !== self::$storage->read('double_quote'), '->write() accepts double quotes in URL');
$profile = new Profile('backslash');
$profile->setUrl('127.0.0.1', 'http://foo.bar/\\');
self::$storage->write($profile);
$this->assertTrue(false !== self::$storage->read('backslash'), '->write() accepts backslash in URL');
$profile = new Profile('comma');
$profile->setUrl('127.0.0.1', 'http://foo.bar/,');
self::$storage->write($profile);
$this->assertTrue(false !== self::$storage->read('comma'), '->write() accepts comma in URL');
}
public function testStoreDuplicateToken()
{
$profile = new Profile('token');
$this->assertTrue(true === self::$storage->write($profile), '->write() returns true when the token is unique');
$this->assertTrue(false === self::$storage->write($profile), '->write() return false when the token is already present in the DB');
}
public function testRetrieveByIp()
{
$profile = new Profile('token');
$profile->setIp('127.0.0.1');
self::$storage->write($profile);
$this->assertEquals(count(self::$storage->find('127.0.0.1', '', 10)), 1, '->find() retrieve a record by IP');
$this->assertEquals(count(self::$storage->find('127.0.%.1', '', 10)), 0, '->find() does not interpret a "%" as a wildcard in the IP');
$this->assertEquals(count(self::$storage->find('127.0._.1', '', 10)), 0, '->find() does not interpret a "_" as a wildcard in the IP');
}
public function testRetrieveByUrl()
{
$profile = new Profile('simple_quote');
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo.bar/\'');
self::$storage->write($profile);
$profile = new Profile('double_quote');
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo.bar/"');
self::$storage->write($profile);
$profile = new Profile('backslash');
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo\\bar/');
self::$storage->write($profile);
$profile = new Profile('percent');
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo.bar/%');
self::$storage->write($profile);
$profile = new Profile('underscore');
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo.bar/_');
self::$storage->write($profile);
$profile = new Profile('semicolon');
$profile->setIp('127.0.0.1');
$profile->setUrl('http://foo.bar/;');
self::$storage->write($profile);
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo.bar/\'', 10)), 1, '->find() accepts single quotes in URLs');
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo.bar/"', 10)), 1, '->find() accepts double quotes in URLs');
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo\\bar/', 10)), 1, '->find() accepts backslash in URLs');
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo.bar/;', 10)), 1, '->find() accepts semicolon in URLs');
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo.bar/%', 10)), 1, '->find() does not interpret a "%" as a wildcard in the URL');
$this->assertEquals(count(self::$storage->find('127.0.0.1', 'http://foo.bar/_', 10)), 1, '->find() does not interpret a "_" as a wildcard in the URL');
}
}