Commits
-------
a0329c3 Added lifetime/cleanup support.
365e73a Fixed the find() method and changed the way the profile data is stored.
beeec5e Allow socket dsn (for example mongodb:///tmp/mongodb-27017.sock).
218eaba Fixed storage of time value.
85c3806 Added support for sorting by time like other profiler storage implementations.
73692c6 Fixed MongoDbProfilerStorage::find() when passing empty parameters.
4cd2dec Use token as identifier to make usage of the automatically created index.
Discussion
----------
[2.1] [HttpKernel] MongoDB profiler updates
I fixed one issue within the MongoDbProfilerStorage::find() function and made some more changes.
---------------------------------------------------------------------------
by snc at 2011/09/11 02:28:35 -0700
Please don't merge this in yet. There are some more commits pending...
---------------------------------------------------------------------------
by fabpot at 2011/09/14 01:07:39 -0700
@snc: is it ready for a merge?
---------------------------------------------------------------------------
by snc at 2011/09/14 01:20:32 -0700
Unfortunately not... while testing I found out that the currently merged in implementation does not work completely. The web profiler search function errors because the find function returns the wrong data. I fixed this already but now I have some strange "maximum function nesting level reached" errors when viewing profiles with children. I will work on it later today.
---------------------------------------------------------------------------
by snc at 2011/09/14 13:27:50 -0700
Now only one thing is missing... the generated container code looks like this:
`$this->services['profiler'] = $instance = new \Symfony\Component\HttpKernel\Profiler\Profiler(new \Symfony\Component\HttpKernel\Profiler\MongoDbProfilerStorage('mongodb://localhost/sf2-mongo-profiler/profiler', '', '', 86400), $a);`
The current constructor only uses the first parameter (dsn). What about the username, password and lifetime? Username and passwort can already be passed via the dsn... but the lifetime feature is not part of the interface... should I implement it?
---------------------------------------------------------------------------
by fabpot at 2011/09/15 11:03:02 -0700
The `lifetime` is used to cleanup the database (see https://github.com/symfony/symfony/blob/master/src/Symfony/Component/HttpKernel/Profiler/PdoProfilerStorage.php#L136). So, it should probably be implemented for MongoDB as well (but it can probably be done in another PR).
---------------------------------------------------------------------------
by snc at 2011/09/19 13:42:52 -0700
Sorry for the delay, lifetime support is now implemented. What do you think about an AbstractProfilerStorageTest class to share some testing code between the different implementations (of cause in a separate PR)?
* 2.0:
[HttpKernel] fixed typo
fixed previous merge, done the same change to other occurences
fixes usage of mb_*
Profiler session import fixed.
[Process] workaround a faulty implementation of is_executable on Windows
Swedish translation fix.
[Locale] Fix#2179 StubIntlDateFormatter support yy format
Fixed fourth argument of Filesystem->mirror()
Commits
-------
41b7a19 Updated the tests so that tests will be marked as skipped when there is no MongoDB server present!
233c7db Updated the code to follow the symfony coding standards
7b24de5 Updated the code to follow the symfony coding standard using stof his remarks
fbcbdde - Fixed a small bug - Updated some phpdoc
00fdfec Added a MongoDbProfilerStorage engine
Discussion
----------
[2.1] [HttpKernel] MongoDb storage for Profiler
As a documentbased database like MongoDB is [supposedly fantastic in logging](http://blog.mongodb.org/post/172254834/mongodb-is-fantastic-for-logging) I implemented a storage engine for the profiler that should enable us to use this database as storage for this.
Activate it using this way:
framework:
profiler:
dsn: mongodb://user:pass@location/database/collection
---------------------------------------------------------------------------
by stof at 2011/07/24 11:23:06 -0700
btw, the MongoDB session storage has already be rejected from the core so this should probably be moved to the DoctrineMongoDBBundle (even if it uses only the PHP extension and not Doctrine). @fabpot thoughts about this ?
---------------------------------------------------------------------------
by Wotre at 2011/07/24 11:52:56 -0700
Just my personal opinion, if it is prefered that way I will move this into the DoctrineMongoDBBundle.
While it is reasonable to bundle all Mongo related things together, I do believe that in the case of logging we want to avoid as many depencies as possible. Some exceptions can occur pretty early inside the framework, and it would be a shame if those aren't logged because this layer is written on top of doctrine. I'm not exactly familliar enough with the symfony internals as I only started using it a few days ago, but I can imagine that this can make a difference with some exceptions.
---------------------------------------------------------------------------
by stof at 2011/07/24 11:59:10 -0700
I don't ask you to use Doctrine in this code. It is fine to use the extension directly if it is enough.
Btw, the profiler is *not* used early. :)
---------------------------------------------------------------------------
by Wotre at 2011/07/26 10:45:05 -0700
So... Any final remark whether this should be moved to [the DoctrineMongoDBBundle](https://github.com/symfony/DoctrineMongoDBBundle) or not?
If it has to be moved, any comment on where in that bundle this should be put?
Also, if it has to be moved, how can we arrange the configuration using DI? Currently I've put a line in the FrameworkExtension file to use this engine for everything with a $dsn starting with mongodb; I imagine this kind of ugly depency can't really exist between the FrameworkBundle and another one.
Although it seems completely illogical to me, I will move it, but I do need some directions on how to elegantly do this...
---------------------------------------------------------------------------
by stof at 2011/07/26 11:03:04 -0700
@fabpot what do you think ?
---------------------------------------------------------------------------
by stof at 2011/09/04 01:28:48 -0700
@fabpot what do you think about the place where this should be done ?
Commits
-------
9f0bd03 [HttpKernel] Update tests for FileProfilerStorage
b7032bc [HttpKernel] Update FileProfileStorage to search from EOF
188a5fa [HttpKernel] Override the existing tokens in FileProfilerStorage
b1b1424 [HttpKernel] Delete folders in the profiler cache
88bc3ec [HttpKernel] Fixes standards of FileProfilerStorage
affe66c Merge remote-tracking branch 'origin/master' into new-profiler-storage
ea916c3 [HttpKernel] Coding convention for the file profiler storage
9ae2c8d [HttpKernel] CS in file storage
b415efd [HttpKernel] Add a test for semicolon in file storage test
1c1215f [HttpKernel] Use subfolders for better storage in file storage of profiler
4b1dc1f [HttpKernel] Fix the folder attribute of file storage to private
70f73e1 [HttpKernel] Fix tests for the file storage of profiler
d5313d9 [HttpKernel] Add tests for the file profiler storage
09fc0a2 [HttpKernel] Add Symfony credits to the file storage class for the profiler
d1d5892 [HttpKernel] Finalize the file storage for the profiler
2f65cf2 Add POC for file storage system
Discussion
----------
[2.1] [HttpKernel] File storage for profiler
Symfony2 has some problems when dealing with multiple concurrency queries in the SQLite storage, resulting in a timeout error or terrible lack.
I've implemented after discussions with @fabpot a filesystem storage.
Enable it in your project with :
framework:
profiler:
dsn: "file:%kernel.cache_dir%/profiler"
I also studied the possibility to store only big data string in files and rest in the SQLite, but not concluant.
Results of my measures (4 concurrency, 120 total) :
* SQLite with data : 1057ms
* SQLite without data : 615ms
* MySQL : 40ms
* This File storage : 54ms
---------------------------------------------------------------------------
by alexandresalome at 2011/07/22 12:01:10 -0700
An idea for the find method : a csv file containing ip;url;token
The iteration could be done over a big file, without loading the whole file in memory.
---------------------------------------------------------------------------
by alexandresalome at 2011/07/23 14:22:32 -0700
OK new version, with as explained previously : a CSV file containing the index + file for each profile.
The speed is similar to the speed of MySQL, and no memory overhead should occur with this solution.
---------------------------------------------------------------------------
by alexandresalome at 2011/07/23 14:37:14 -0700
Hm... Created tests, duplicated from SqliteProfilerStorageTest.
Any idea on how to put this code in common ? Is it usual to create a base class for 2 tests ?
---------------------------------------------------------------------------
by alexandresalome at 2011/07/23 14:48:39 -0700
Just tested with 24.000 requests, the 24.001'th request still takes less than 50ms to execute.
The index file is about 2Mb, and iterating the whole file is fast.
---------------------------------------------------------------------------
by alexandresalome at 2011/07/23 14:53:19 -0700
I've filled the file with 120Mb of data, requests are still less than 50ms for executing.
Iterating the index takes more than 30s (so it crashed), but it's because of the amount of lines. 30 seconds = 1,400,000 lines in this computer. The file = 1,500,000 lines
---------------------------------------------------------------------------
by alexandresalome at 2011/07/23 14:56:54 -0700
I've tested it with Linux, is someone can test with Windows
---------------------------------------------------------------------------
by stloyd at 2011/07/24 00:32:32 -0700
IMO to speedup it a bit more and not end up with "crash" (to not end with "limit" of files per directory, also to many files in dir slow down every OS) you should use same method to write as Twig, split up files in to directories. If you do this you can speed up index more, because you can create smaller one per directory.
Also you should fix CS (coding standards).
---------------------------------------------------------------------------
by stloyd at 2011/07/24 02:10:20 -0700
Tested on Win 7, seems ok. Similar speed to sqlite, dunno why ;-) but used a bit less of memory.
---------------------------------------------------------------------------
by alexandresalome at 2011/07/24 02:13:21 -0700
Did you tried with concurrent requests ? It makes sense when you use assetic
and your browser hits the application 4 times simultaneously for CSS
generation
---------------------------------------------------------------------------
by alexandresalome at 2011/07/24 02:17:23 -0700
I used Apache Benchmark for producing results :
ab -c 4 -n 120 URL
---------------------------------------------------------------------------
by alexandresalome at 2011/07/24 02:56:55 -0700
OK I used subfolders, based on last characters (because the first part of token is mostly the same between queries.
---------------------------------------------------------------------------
by stof at 2011/09/04 01:27:15 -0700
@fabpot any news about it ? Can it be merged ?
* The register() method on all listeners has been removed
* Instead, the information is now put directly in the DIC tag
For instance, a listener on core.request had this method:
public function register(EventDispatcher $dispatcher, $priority = 0)
{
$dispatcher->connect('core.response', array($this, 'filter'), $priority);
}
And this tag in the DIC configuration:
<tag name="kernel.listener" />
Now, it only has the following configuration:
<tag name="kernel.listener" event="core.response" method="filter" priority="0" />
The event and method attributes are now mandatory.
When an object has a "main" many relation with related "things" (objects,
parameters, ...), the method names are normalized:
* get()
* set()
* all()
* replace()
* remove()
* clear()
* isEmpty()
* add()
* register()
* count()
* keys()
The classes below follow this method naming convention:
* BrowserKit\CookieJar -> Cookie
* BrowserKit\History -> Request
* Console\Application -> Command
* Console\Application\Helper\HelperSet -> HelperInterface
* DependencyInjection\Container -> services
* DependencyInjection\ContainerBuilder -> services
* DependencyInjection\ParameterBag\ParameterBag -> parameters
* DependencyInjection\ParameterBag\FrozenParameterBag -> parameters
* DomCrawler\Form -> FormField
* EventDispatcher\Event -> parameters
* Form\FieldGroup -> Field
* HttpFoundation\HeaderBag -> headers
* HttpFoundation\ParameterBag -> parameters
* HttpFoundation\Session -> attributes
* HttpKernel\Profiler\Profiler -> DataCollectorInterface
* Routing\RouteCollection -> Route
* Security\Authentication\AuthenticationProviderManager -> AuthenticationProviderInterface
* Templating\Engine -> HelperInterface
* Translation\MessageCatalogue -> messages
The usage of these methods are only allowed when it is clear that there is a
main relation:
* a CookieJar has many Cookies;
* a Container has many services and many parameters (as services is the main
relation, we use the naming convention for this relation);
* a Console Input has many arguments and many options. There is no "main"
relation, and so the naming convention does not apply.
For many relations where the convention does not apply, the following methods
must be used instead (where XXX is the name of the related thing):
* get() -> getXXX()
* set() -> setXXX()
* all() -> getXXXs()
* replace() -> setXXXs()
* remove() -> removeXXX()
* clear() -> clearXXX()
* isEmpty() -> isEmptyXXX()
* add() -> addXXX()
* register() -> registerXXX()
* count() -> countXXX()
* keys()
