Skip to content

Commit 39d2136

Browse files
mpdudegreg0ire
mpdude
and
greg0ire
authored
Fix different first/max result values taking up query cache space (#11188)
* Add a test covering the #11112 issue * Add new OutputWalker and SqlFinalizer interfaces * Add a SingleSelectSqlFinalizer that can take care of adding offset/limit as well as locking mode statements to a given SQL query. Add a FinalizedSelectExecutor that executes given, finalized SQL statements. * In SqlWalker, split SQL query generation into the two parts that shall happen before and after the finalization phase. Move the part that generates "pre-finalization" SQL into a dedicated method. Use a side channel in SingleSelectSqlFinalizer to access the "finalization" logic and avoid duplication. * Fix CS violations * Skip the GH11112 test while applying refactorings * Avoid a Psalm complaint due to invalid (?) docblock syntax * Establish alternate code path - queries can obtain the sql executor through the finalizer, parser knows about output walkers yielding finalizers * Remove a possibly premature comment * Re-enable the #11112 test * Fix CS * Make RootTypeWalker inherit from SqlOutputWalker so it becomes finalizer-aware * Update QueryCacheTest, since first/max results no longer need extra cache entries * Fix ParserResultSerializationTest by forcing the parser to produce a ParserResult of the old kind (with the executor already constructed) * Fix WhereInWalkerTest * Update lib/Doctrine/ORM/Query/Exec/PreparedExecutorFinalizer.php Co-authored-by: Grégoire Paris <postmaster@greg0ire.fr> * Fix tests * Fix a Psalm complaint * Fix a test * Fix CS * Make the NullSqlWalker an instance of SqlOutputWalker * Avoid multiple cache entries caused by LimitSubqueryOutputWalker * Fix Psalm complaints * Fix static analysis complaints * Remove experimental code that I committed accidentally * Remove unnecessary baseline entry * Make AddUnknownQueryComponentWalker subclass SqlOutputWalker That way, we have no remaining classes in the codebase subclassing SqlWalker but not SqlOutputWalker * Use more expressive exception classes * Add a deprecation message * Move SqlExecutor creation to ParserResult, to minimize public methods available on it * Avoid keeping the SqlExecutor in the Query, since it must be generated just in time (e. g. in case Query parameters change) * Address PHPStan complaints * Fix tests * Small refactorings * Add an upgrade notice * Small refactorings * Update the Psalm baseline * Add a missing namespace import * Update Psalm baseline * Fix CS * Fix Psalm baseline --------- Co-authored-by: Grégoire Paris <postmaster@greg0ire.fr>
1 parent bea454e commit 39d2136

23 files changed

+591
-104
lines changed

‎UPGRADE.md

+9
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
11
# Upgrade to 2.20
22

3+
## Add `Doctrine\ORM\Query\OutputWalker` interface, deprecate `Doctrine\ORM\Query\SqlWalker::getExecutor()`
4+
5+
Output walkers should implement the new `\Doctrine\ORM\Query\OutputWalker` interface and create
6+
`Doctrine\ORM\Query\Exec\SqlFinalizer` instances instead of `Doctrine\ORM\Query\Exec\AbstractSqlExecutor`s.
7+
The output walker must not base its workings on the query `firstResult`/`maxResult` values, so that the
8+
`SqlFinalizer` can be kept in the query cache and used regardless of the actual `firstResult`/`maxResult` values.
9+
Any operation dependent on `firstResult`/`maxResult` should take place within the `SqlFinalizer::createExecutor()`
10+
method. Details can be found at https://github.com/doctrine/orm/pull/11188.
11+
312
## Explictly forbid property hooks
413

514
Property hooks are not supported yet by Doctrine ORM. Until support is added,

‎psalm-baseline.xml

+9-5
Original file line numberDiff line numberDiff line change
@@ -1868,6 +1868,12 @@
18681868
<code><![CDATA[$this->_sqlStatements = &$this->sqlStatements]]></code>
18691869
</UnsupportedPropertyReferenceUsage>
18701870
</file>
1871+
<file src="src/Query/Exec/FinalizedSelectExecutor.php">
1872+
<PropertyNotSetInConstructor>
1873+
<code><![CDATA[FinalizedSelectExecutor]]></code>
1874+
<code><![CDATA[FinalizedSelectExecutor]]></code>
1875+
</PropertyNotSetInConstructor>
1876+
</file>
18711877
<file src="src/Query/Exec/MultiTableDeleteExecutor.php">
18721878
<InvalidReturnStatement>
18731879
<code><![CDATA[$numDeleted]]></code>
@@ -1996,6 +2002,9 @@
19962002
<ArgumentTypeCoercion>
19972003
<code><![CDATA[$stringPattern]]></code>
19982004
</ArgumentTypeCoercion>
2005+
<DeprecatedMethod>
2006+
<code><![CDATA[setSqlExecutor]]></code>
2007+
</DeprecatedMethod>
19992008
<InvalidNullableReturnType>
20002009
<code><![CDATA[AST\SelectStatement|AST\UpdateStatement|AST\DeleteStatement]]></code>
20012010
</InvalidNullableReturnType>
@@ -2057,11 +2066,6 @@
20572066
<code><![CDATA[$token === TokenType::T_IDENTIFIER]]></code>
20582067
</RedundantConditionGivenDocblockType>
20592068
</file>
2060-
<file src="src/Query/ParserResult.php">
2061-
<PropertyNotSetInConstructor>
2062-
<code><![CDATA[$sqlExecutor]]></code>
2063-
</PropertyNotSetInConstructor>
2064-
</file>
20652069
<file src="src/Query/QueryExpressionVisitor.php">
20662070
<InvalidReturnStatement>
20672071
<code><![CDATA[new ArrayCollection($this->parameters)]]></code>

‎src/Query.php

+31-3
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@
1818
use Doctrine\ORM\Query\AST\SelectStatement;
1919
use Doctrine\ORM\Query\AST\UpdateStatement;
2020
use Doctrine\ORM\Query\Exec\AbstractSqlExecutor;
21+
use Doctrine\ORM\Query\Exec\SqlFinalizer;
22+
use Doctrine\ORM\Query\OutputWalker;
2123
use Doctrine\ORM\Query\Parameter;
2224
use Doctrine\ORM\Query\ParameterTypeInferer;
2325
use Doctrine\ORM\Query\Parser;
@@ -33,6 +35,7 @@
3335
use function count;
3436
use function get_debug_type;
3537
use function in_array;
38+
use function is_a;
3639
use function is_int;
3740
use function ksort;
3841
use function md5;
@@ -196,7 +199,7 @@ class Query extends AbstractQuery
196199
*/
197200
public function getSQL()
198201
{
199-
return $this->parse()->getSqlExecutor()->getSqlStatements();
202+
return $this->getSqlExecutor()->getSqlStatements();
200203
}
201204

202205
/**
@@ -285,7 +288,7 @@ private function parse(): ParserResult
285288
*/
286289
protected function _doExecute()
287290
{
288-
$executor = $this->parse()->getSqlExecutor();
291+
$executor = $this->getSqlExecutor();
289292

290293
if ($this->_queryCacheProfile) {
291294
$executor->setQueryCacheProfile($this->_queryCacheProfile);
@@ -813,11 +816,31 @@ protected function getQueryCacheId(): string
813816
{
814817
ksort($this->_hints);
815818

819+
if (! $this->hasHint(self::HINT_CUSTOM_OUTPUT_WALKER)) {
820+
// Assume Parser will create the SqlOutputWalker; save is_a call, which might trigger a class load
821+
$firstAndMaxResult = '';
822+
} else {
823+
$outputWalkerClass = $this->getHint(self::HINT_CUSTOM_OUTPUT_WALKER);
824+
if (is_a($outputWalkerClass, OutputWalker::class, true)) {
825+
$firstAndMaxResult = '';
826+
} else {
827+
Deprecation::trigger(
828+
'doctrine/orm',
829+
'https://github.com/doctrine/orm/pull/11188/',
830+
'Your output walker class %s should implement %s in order to provide a %s. This also means the output walker should not use the query firstResult/maxResult values, which should be read from the query by the SqlFinalizer only.',
831+
$outputWalkerClass,
832+
OutputWalker::class,
833+
SqlFinalizer::class
834+
);
835+
$firstAndMaxResult = '&firstResult=' . $this->firstResult . '&maxResult=' . $this->maxResults;
836+
}
837+
}
838+
816839
return md5(
817840
$this->getDQL() . serialize($this->_hints) .
818841
'&platform=' . get_debug_type($this->getEntityManager()->getConnection()->getDatabasePlatform()) .
819842
($this->_em->hasFilters() ? $this->_em->getFilters()->getHash() : '') .
820-
'&firstResult=' . $this->firstResult . '&maxResult=' . $this->maxResults .
843+
$firstAndMaxResult .
821844
'&hydrationMode=' . $this->_hydrationMode . '&types=' . serialize($this->parsedTypes) . 'DOCTRINE_QUERY_CACHE_SALT'
822845
);
823846
}
@@ -836,4 +859,9 @@ public function __clone()
836859

837860
$this->state = self::STATE_DIRTY;
838861
}
862+
863+
private function getSqlExecutor(): AbstractSqlExecutor
864+
{
865+
return $this->parse()->prepareSqlExecutor($this);
866+
}
839867
}

‎src/Query/Exec/FinalizedSelectExecutor.php

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace Doctrine\ORM\Query\Exec;
6+
7+
use Doctrine\DBAL\Connection;
8+
use Doctrine\DBAL\Result;
9+
use Doctrine\DBAL\Types\Type;
10+
11+
/**
12+
* SQL executor for a given, final, single SELECT SQL query
13+
*
14+
* @method string getSqlStatements()
15+
*/
16+
class FinalizedSelectExecutor extends AbstractSqlExecutor
17+
{
18+
public function __construct(string $sql)
19+
{
20+
parent::__construct();
21+
22+
$this->sqlStatements = $sql;
23+
}
24+
25+
/**
26+
* @param list<mixed>|array<string, mixed> $params
27+
* @param array<int, int|string|Type|null>|array<string, int|string|Type|null> $types
28+
*/
29+
public function execute(Connection $conn, array $params, array $types): Result
30+
{
31+
return $conn->executeQuery($this->getSqlStatements(), $params, $types, $this->queryCacheProfile);
32+
}
33+
}

‎src/Query/Exec/PreparedExecutorFinalizer.php

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace Doctrine\ORM\Query\Exec;
6+
7+
use Doctrine\ORM\Query;
8+
9+
/**
10+
* PreparedExecutorFinalizer is a wrapper for the SQL finalization
11+
* phase that does nothing - it is constructed with the sql executor
12+
* already.
13+
*/
14+
final class PreparedExecutorFinalizer implements SqlFinalizer
15+
{
16+
/** @var AbstractSqlExecutor */
17+
private $executor;
18+
19+
public function __construct(AbstractSqlExecutor $exeutor)
20+
{
21+
$this->executor = $exeutor;
22+
}
23+
24+
public function createExecutor(Query $query): AbstractSqlExecutor
25+
{
26+
return $this->executor;
27+
}
28+
}

‎src/Query/Exec/SingleSelectSqlFinalizer.php

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace Doctrine\ORM\Query\Exec;
6+
7+
use Doctrine\DBAL\LockMode;
8+
use Doctrine\ORM\Query;
9+
use Doctrine\ORM\Query\QueryException;
10+
use Doctrine\ORM\Utility\LockSqlHelper;
11+
12+
/**
13+
* SingleSelectSqlFinalizer finalizes a given SQL query by applying
14+
* the query's firstResult/maxResult values as well as extra read lock/write lock
15+
* statements, both through the platform-specific methods.
16+
*
17+
* The resulting, "finalized" SQL is passed to a FinalizedSelectExecutor.
18+
*/
19+
class SingleSelectSqlFinalizer implements SqlFinalizer
20+
{
21+
use LockSqlHelper;
22+
23+
/** @var string */
24+
private $sql;
25+
26+
public function __construct(string $sql)
27+
{
28+
$this->sql = $sql;
29+
}
30+
31+
/**
32+
* This method exists temporarily to support old SqlWalker interfaces.
33+
*
34+
* @internal
35+
*
36+
* @psalm-internal Doctrine\ORM
37+
*/
38+
public function finalizeSql(Query $query): string
39+
{
40+
$platform = $query->getEntityManager()->getConnection()->getDatabasePlatform();
41+
42+
$sql = $platform->modifyLimitQuery($this->sql, $query->getMaxResults(), $query->getFirstResult());
43+
44+
$lockMode = $query->getHint(Query::HINT_LOCK_MODE) ?: LockMode::NONE;
45+
46+
if ($lockMode !== LockMode::NONE && $lockMode !== LockMode::OPTIMISTIC && $lockMode !== LockMode::PESSIMISTIC_READ && $lockMode !== LockMode::PESSIMISTIC_WRITE) {
47+
throw QueryException::invalidLockMode();
48+
}
49+
50+
if ($lockMode === LockMode::PESSIMISTIC_READ) {
51+
$sql .= ' ' . $this->getReadLockSQL($platform);
52+
} elseif ($lockMode === LockMode::PESSIMISTIC_WRITE) {
53+
$sql .= ' ' . $this->getWriteLockSQL($platform);
54+
}
55+
56+
return $sql;
57+
}
58+
59+
/** @return FinalizedSelectExecutor */
60+
public function createExecutor(Query $query): AbstractSqlExecutor
61+
{
62+
return new FinalizedSelectExecutor($this->finalizeSql($query));
63+
}
64+
}

‎src/Query/Exec/SqlFinalizer.php

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace Doctrine\ORM\Query\Exec;
6+
7+
use Doctrine\ORM\Query;
8+
9+
/**
10+
* SqlFinalizers are created by OutputWalkers that traversed the DQL AST.
11+
* The SqlFinalizer instance can be kept in the query cache and re-used
12+
* at a later time.
13+
*
14+
* Once the SqlFinalizer has been created or retrieved from the query cache,
15+
* it receives the Query object again in order to yield the AbstractSqlExecutor
16+
* that will then be used to execute the query.
17+
*
18+
* The SqlFinalizer may assume that the DQL that was used to build the AST
19+
* and run the OutputWalker (which created the SqlFinalizer) is equivalent to
20+
* the query that will be passed to the createExecutor() method. Potential differences
21+
* are the parameter values or firstResult/maxResult settings.
22+
*/
23+
interface SqlFinalizer
24+
{
25+
public function createExecutor(Query $query): AbstractSqlExecutor;
26+
}

‎src/Query/OutputWalker.php

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?php
2+
3+
declare(strict_types=1);
4+
5+
namespace Doctrine\ORM\Query;
6+
7+
use Doctrine\ORM\Query\Exec\SqlFinalizer;
8+
9+
/**
10+
* Interface for output walkers
11+
*
12+
* Output walkers, like tree walkers, can traverse the DQL AST to perform
13+
* their purpose.
14+
*
15+
* The goal of an OutputWalker is to ultimately provide the SqlFinalizer
16+
* which produces the final, executable SQL statement in a "finalization" phase.
17+
*
18+
* It must be possible to use the same SqlFinalizer for Queries with different
19+
* firstResult/maxResult values. In other words, SQL produced by the
20+
* output walker should not depend on those values, and any SQL generation/modification
21+
* specific to them should happen in the finalizer's `\Doctrine\ORM\Query\Exec\SqlFinalizer::createExecutor()`
22+
* method instead.
23+
*/
24+
interface OutputWalker
25+
{
26+
/** @param AST\DeleteStatement|AST\UpdateStatement|AST\SelectStatement $AST */
27+
public function getFinalizer($AST): SqlFinalizer;
28+
}

‎src/Query/Parser.php

+19-3
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
use Doctrine\ORM\Mapping\ClassMetadata;
1111
use Doctrine\ORM\Query;
1212
use Doctrine\ORM\Query\AST\Functions;
13+
use Doctrine\ORM\Query\Exec\SqlFinalizer;
1314
use LogicException;
1415
use ReflectionClass;
1516

@@ -396,11 +397,26 @@ public function parse()
396397
$this->queryComponents = $treeWalkerChain->getQueryComponents();
397398
}
398399

399-
$outputWalkerClass = $this->customOutputWalker ?: SqlWalker::class;
400+
$outputWalkerClass = $this->customOutputWalker ?: SqlOutputWalker::class;
400401
$outputWalker = new $outputWalkerClass($this->query, $this->parserResult, $this->queryComponents);
401402

402-
// Assign an SQL executor to the parser result
403-
$this->parserResult->setSqlExecutor($outputWalker->getExecutor($AST));
403+
if ($outputWalker instanceof OutputWalker) {
404+
$finalizer = $outputWalker->getFinalizer($AST);
405+
$this->parserResult->setSqlFinalizer($finalizer);
406+
} else {
407+
Deprecation::trigger(
408+
'doctrine/orm',
409+
'https://github.com/doctrine/orm/pull/11188/',
410+
'Your output walker class %s should implement %s in order to provide a %s. This also means the output walker should not use the query firstResult/maxResult values, which should be read from the query by the SqlFinalizer only.',
411+
$outputWalkerClass,
412+
OutputWalker::class,
413+
SqlFinalizer::class
414+
);
415+
// @phpstan-ignore method.deprecated
416+
$executor = $outputWalker->getExecutor($AST);
417+
// @phpstan-ignore method.deprecated
418+
$this->parserResult->setSqlExecutor($executor);
419+
}
404420

405421
return $this->parserResult;
406422
}

0 commit comments

Comments
 (0)