Can Kedro handle templated queries?

[atercygnus]

I apologize if this feature is already implemented, but I can't find one.

Need to have SQLQueryDataSet with parameters.
Something like:

**catalog.yml:**

foo_dataset:
    type: SQLQueryDataSet
    credentials: foo_cred
    sql:
        select id, ... from foo_table where id in ({%ids})

**notebook or .py file:**

ids_tofetch = [...]
df = context.catalog.load('foo_dataset', {'ids':ids_tofetch})

Can this be done for now?

[yetudada]

Hi @atercygnus! Thanks for raising this query. A templated query, or in your case, a templated catalog.yml is possible. Here are the docs for this: https://kedro.readthedocs.io/en/latest/04_user_guide/03_configuration.html#templating-configuration. However, this would still require you to create a globals.yml in conf/ to host your ids_tofetch. Is this possible in your use case?

We actually find that Stack Overflow might be better suited to these kind of questions, so please do post there next, tag your questions with kedro and we'll find them.

[atercygnus]

Hi @yetudada! Thanks for answering!

This is nice, and solves a part of my problem, but in many cases sql queries are too complicated to be a single query, and it's convenient to fetch parts of the data separately, and then merge them in python. In this case it is essential to have templated queries without using additional file globals.yml.

[lorenabalan]

Closing this issue as stale, but please feel free to raise a new one if you encounter other problems. As Yetu noted, TemplatedConfigLoader could be helpful here. Otherwise maybe a local customisation for the SQLDataSet is the best guess.