Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

query_data_frame with many columns throws Pandas PerformanceWarning #347

Closed
sjamesc opened this issue Oct 17, 2021 · 1 comment · Fixed by #348
Closed

query_data_frame with many columns throws Pandas PerformanceWarning #347

sjamesc opened this issue Oct 17, 2021 · 1 comment · Fixed by #348
Labels
enhancement New feature or request
Milestone

Comments

@sjamesc
Copy link

sjamesc commented Oct 17, 2021

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

  1. Write a flux query with a pivot that results in many columns being returned (i.e. 100+)
  2. Use client.query_api().query_data_frame(query, org=org) to query as a Pandas DataFrame

Expected behavior:
The code should run without issue, similar to how it runs fine without the pivot (so many rows, not many columns).

Actual behavior:
Pandas warning is thrown:

/opt/conda/envs/trading/lib/python3.9/site-packages/influxdb_client/client/flux_csv_parser.py:138: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use newframe = frame.copy()
self._data_frame[column.label] = column.default_value

Specifications:

  • Client Version: 1.21.0
  • Pandas Version: 1.3.3
  • InfluxDB Version: 2.0.9
  • Platform: Python 3.9
@bednar
Copy link
Contributor

bednar commented Oct 18, 2021

Hi @sjamesc,

thanks for using our client.

We will take a look.

Regards

@bednar bednar added the enhancement New feature or request label Oct 19, 2021
@bednar bednar added this to the 1.22.0 milestone Oct 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants