Power spectrum responses for SSC #1134

RyoTerasawa · 2023-11-06T00:23:10Z

In pyccl/pkresponse.py, I implemented the functions to calculate power spectrum responses to the super-survey modes which are responsible for super-sample covariance (SSC), based on arXiv:2310.13330.
We approximate the power spectrum growth response to the super-survey modes by its growth response to the Hubble parameter.

For the galaxy-matter and galaxy-auto power spectra, the halo statistics (halo-matter/halo-auto power spectrum, mass function) are calculated by DarkEmulator.

coveralls · 2023-12-04T08:34:58Z

Pull Request Test Coverage Report for Build 12132306089

Details

304 of 346 (87.86%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.5%) to 96.946%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pyccl/pkresponse.py	303	345	87.83%

Totals
Change from base Build 11958349869:	-0.5%
Covered Lines:	6824
Relevant Lines:	7039

💛 - Coveralls

damonge · 2024-02-21T08:35:24Z

@RyoTerasawa can I check what the status of this PR is?

RyoTerasawa · 2024-02-21T11:17:50Z

I am waiting for someone to review the code. I haven't asked for a specific person yet, but maybe I should ask a member of the MCPCov group to review it. I am happy to explain the details if needed. I am also working on writing the unit test and benchmark test to be included in this PR.

carlosggarcia · 2024-04-08T14:47:11Z

@RyoTerasawa @YueNan-c, maybe you want to also implement the DarkEmulator here: https://github.com/LSSTDESC/CCL/blob/master/pyccl/emulators/cosmicemu_pk.py.

Are the unit tests and benchmark ready?

damonge · 2024-06-18T08:51:18Z

@RyoTerasawa @carlosggarcia can I check what the status of this is?

RyoTerasawa · 2024-06-22T12:57:20Z

@carlosggarcia @damonge
Sorry for the late reply. I just added the unit tests and benchmark, and this pull request is ready to be reviewed.

carlosggarcia

Ok. I first round of comments. Some suggestions:

Add unit tests per function/method declared to be sure they are doing what they should
I would unify the .npy files in a single .npz file. Also, it seems that you are only using the z=0 case, I'd be great if you can compare all redshift, since you have it.
Add References to the equations in your paper for later reference (and to make my life easier when reviewing the PR)
@damonge, there are integrals and biases computed that I think they might be obtained from the current CCL implementation. Can you have a look? The are in pkresponse.py, where I pinged you, too.

In general, it looks good, though! Good work!

pyccl/tests/test_pkresponse.py

pyccl/pkresponse.py

pyccl/tests/test_pkresponse.py

pyccl/pkresponse.py

carlosggarcia · 2024-07-03T14:21:11Z

pyccl/pkresponse.py

+            / ng
+        )
+
+        bgE2 = (


@damonge , can this be done with the current methods of the halo model?

Same with the other integrals

I think so. @RyoTerasawa can you check:

CCL/pyccl/halos/halo_model.py

Line 108 in 5273ca6

def _integrate_over_mf(self, array_2):

and

CCL/pyccl/halos/halo_model.py

Line 113 in 5273ca6

def _integrate_over_mbf(self, array_2):

and

CCL/pyccl/halos/halo_model.py

Line 118 in 5273ca6

def integrate_over_massfunc(self, func, cosmo, a):

?

I changed the pkresponse.py, and now all the integrals of the mass function are using _integrate_over_mf or _integrate_over_mbf.

benchmarks/test_pkresponse.py

damonge · 2024-11-13T10:06:32Z

@carlosggarcia @RyoTerasawa can I check what the status of this PR is?

RyoTerasawa · 2024-11-15T06:03:31Z

@carlosggarcia @RyoTerasawa can I check what the status of this PR is?

I am currently working on addressing Carlos's comments.

RyoTerasawa · 2024-12-03T03:31:06Z

@carlosggarcia Thank you for the comments and suggestions.
I complete addressing these comments except for the integrals.

Added unit tests per function/method declared to be sure they are doing what they should
Unified the .npy files in a single .npz file and now the benchmark test compares all redshift.
Added References to the equations in the paper.
Have not addressed to the integrals in pkresponse.py

@damonge, there are integrals and biases computed that I think they might be obtained from the current CCL implementation. Can you have a look? The are in pkresponse.py, where I pinged you, too.

damonge · 2024-12-04T09:04:40Z

@RyoTerasawa I responded to Carlos' comment about the integrals. If possible, it'd be good if CCL does all of these consistently.

RyoTerasawa · 2024-12-08T07:35:34Z

@damonge Thank you for the suggestion. I switched to the integral method of pyccl/halos/halo_model.py and it helped to make the code bit simpler.

Merge branch 'master' into SSC

RyoTerasawa · 2024-12-08T08:17:36Z

@carlosggarcia, I complete addressing these comments. Although the build failed at the unit test.

Allowing for cross-correlations in HOD power spectrum. (#1224)

RyoTerasawa · 2025-02-13T23:25:15Z

@carlosggarcia I believe this PR is ready for review. Whenever you have time, I would appreciate your feedback. Thank you!

pyccl/pkresponse.py

carlosggarcia

A few more comment/questions.

I didn't have time to go through all the equation in pkresponse.py but I guess that if the benchmark pass, they will be ok.

The most important comment is the one regarding the benchmarks. It seems to me that the threshold is to high (e.g. 60 sigma away of what you measure in the simulation)

carlosggarcia · 2025-03-08T17:22:26Z

benchmarks/test_pkresponse.py

+    assert np.allclose(
+        Pmm_resp_data[0, indx_mm],
+        generated_Pmm_resp[0],
+        atol=10 * Pmm_resp_err_data[0, indx_mm],


Aren't all the atol a bit too big? this is like 10 sigma, right? And you have others with even larger numbers.

I guess the error bars from the simulations are too strict, probably because we use the pair and fixed initial density method. Instead, I switched to rtol.
The difference between the model and the simulations are typically few ten percents. Higher redshifts of Pgm, Pgg (especially Pgg) have worse accuracy due to the accuracy of darkemulator itself, but they are still a factor of difference at most.

Okay, make sense. Can you add a comment in the tests explaining this? Otherwise we will forget in a few months.

I added above comments in the benchmarks/test_pkrespose.py.

carlosggarcia

All looks good to me. Can you merge the master branch into this branch to see if that solves the issue with the unit tests?
Once the unit test pass, I will approve.

I've left some questions for @damonge regarding the integrals.

carlosggarcia · 2025-03-12T11:06:59Z

pyccl/pkresponse.py

+        nP = nth_mat * np.array(Pth)
+
+        # Eq. A7
+        Pgm = integrate.romb(dprof_dlogM * nP, dx=dlogM, axis=0) / ng


I think this could be done with current halo calculator, @damonge?

I've commented those that I think might use current CCL integrators

carlosggarcia · 2025-03-12T11:09:07Z

pyccl/pkresponse.py

+        Pgg_2h_int = np.array(Pgg_2h_int)
+
+        # Eq. A12
+        _Pgg_2h = integrate.romb(


Maybe this one too? @damonge

carlosggarcia · 2025-03-12T11:11:06Z

pyccl/pkresponse.py

+        G_prof = (
+            +2.0
+            / 3.0
+            * integrate.romb(


Maybe this too? @damonge

RyoTerasawa · 2025-03-12T14:45:10Z

All looks good to me. Can you merge the master branch into this branch to see if that solves the issue with the unit tests? Once the unit test pass, I will approve.

I've left some questions for @damonge regarding the integrals.

Thank you for the through review. Now this SSC branch is up to date with the master branch, and 33 commits ahead of the master branch.
As the unit tests still fail at the point other than my modifications, I will wait for that issue to be resolved.

merge from master branch

RyoTerasawa added 4 commits November 6, 2023 07:19

add darkemulator.py

867e33c

add darkemulator mass function

0df32e9

Linting

a37dcca

delete darkemulator.py

efcdc7f

RyoTerasawa requested a review from carlosggarcia November 6, 2023 00:23

add dependency on dark_emulator

2ee5ba5

This was referenced Dec 4, 2023

Ssc terasawa #991

Closed

SSC for g-g lensing and galaxy clustering using DarkEmulator #1079

Closed

Merge remote-tracking branch 'origin/master' into SSC

9b7450d

RyoTerasawa force-pushed the SSC branch from 43d5eb5 to 9b7450d Compare June 4, 2024 05:27

damonge and others added 6 commits June 18, 2024 09:51

Merge branch 'master' into SSC

f76b69f

try bumping python

4b5f3a6

try george in conda

a382551

Merge branch 'master' into SSC

5f8d35d

add unit test and benchmark test

ca3dcd9

lint

0341150

carlosggarcia requested changes Jul 3, 2024

View reviewed changes

RyoTerasawa and others added 5 commits December 2, 2024 01:58

Merge branch 'master' into SSC

12fe1d1

Add References to the equations

a25862e

switch to MassFuncNishimichi19

08af485

add dependency to extra_parameters

46e92de

make sure the cosmology object is exactly the same

d4a21ca

RyoTerasawa added 6 commits December 2, 2024 07:33

create pkresponse.npz

f7347fe

update benchmark

896f812

add documentation

59ea6ea

make utility functions private and add unit tests for them

a205e5e

update test_pkresponse.py

de64ba3

lint

1928d7c

switched to halo model integral

2925dac

Merge pull request #1217 from LSSTDESC/master

2f5417a

Merge branch 'master' into SSC

RyoTerasawa and others added 5 commits February 10, 2025 21:24

rerun Github jobs

b26134a

Merge remote-tracking branch 'origin/SSC' into SSC

937bf8f

change to Miniforge3

f79e4c8

Update ci.yml

3ed8434

Merge pull request #1225 from LSSTDESC/master

d0e7fbe

Allowing for cross-correlations in HOD power spectrum. (#1224)

carlosggarcia reviewed Mar 8, 2025

View reviewed changes

pyccl/pkresponse.py Outdated Show resolved Hide resolved

carlosggarcia requested changes Mar 8, 2025

View reviewed changes

RyoTerasawa added 2 commits March 9, 2025 05:58

Merge remote-tracking branch 'origin/SSC' into SSC

03845c0

update benchmark

fdd1a9f

carlosggarcia reviewed Mar 12, 2025

View reviewed changes

add comments in benchmark

4a9e8bb

Merge pull request #1239 from LSSTDESC/master

45d8c0f

merge from master branch

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Power spectrum responses for SSC #1134

Power spectrum responses for SSC #1134

RyoTerasawa commented Nov 6, 2023

coveralls commented Dec 4, 2023 •

edited

Loading

damonge commented Feb 21, 2024

RyoTerasawa commented Feb 21, 2024

carlosggarcia commented Apr 8, 2024

damonge commented Jun 18, 2024

RyoTerasawa commented Jun 22, 2024

carlosggarcia left a comment

carlosggarcia Jul 3, 2024

carlosggarcia Jul 3, 2024

damonge Dec 4, 2024

RyoTerasawa Dec 8, 2024

damonge commented Nov 13, 2024

RyoTerasawa commented Nov 15, 2024

RyoTerasawa commented Dec 3, 2024

damonge commented Dec 4, 2024

RyoTerasawa commented Dec 8, 2024

RyoTerasawa commented Dec 8, 2024

RyoTerasawa commented Feb 13, 2025

carlosggarcia left a comment

carlosggarcia Mar 8, 2025

RyoTerasawa Mar 8, 2025

carlosggarcia Mar 12, 2025

RyoTerasawa Mar 12, 2025

carlosggarcia left a comment

carlosggarcia Mar 12, 2025

carlosggarcia Mar 12, 2025

carlosggarcia Mar 12, 2025

RyoTerasawa commented Mar 12, 2025

Power spectrum responses for SSC #1134

Are you sure you want to change the base?

Power spectrum responses for SSC #1134

Conversation

RyoTerasawa commented Nov 6, 2023

coveralls commented Dec 4, 2023 • edited Loading

Pull Request Test Coverage Report for Build 12132306089

Details

💛 - Coveralls

damonge commented Feb 21, 2024

RyoTerasawa commented Feb 21, 2024

carlosggarcia commented Apr 8, 2024

damonge commented Jun 18, 2024

RyoTerasawa commented Jun 22, 2024

carlosggarcia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damonge commented Nov 13, 2024

RyoTerasawa commented Nov 15, 2024

RyoTerasawa commented Dec 3, 2024

damonge commented Dec 4, 2024

RyoTerasawa commented Dec 8, 2024

RyoTerasawa commented Dec 8, 2024

RyoTerasawa commented Feb 13, 2025

carlosggarcia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosggarcia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyoTerasawa commented Mar 12, 2025

coveralls commented Dec 4, 2023 •

edited

Loading