support giving less resources to init containers

[joel-bluedata]

Currently the init container for setting up the PV gets a hardcoded 1 CPU and 512Mi memory.

An annoying-in-some-circumstances effect of this: the reserved resources for a kdcluster pod is the max of the init container request and whatever the pod requests. It's not uncommon for a smaller kdapp (perhaps with numerous kdcluster instances) to want to use less than 1 CPU per pod, but if it uses a PV then it will effectively not be able to use less than 1 CPU per pod because of the init container.

If we move to a lower hardcoded value e.g. 250m, that may be an improvement in many cases... but in other cases this might cause the PV init to run slower for big apps that have a lot of data to copy in that phase. Which is bad!

So we may want to change the hardcoded amounts here, and we almost certainly want the amounts to be user-tunable, in the KD config and/or in the kdcluster spec. Easy enough to implement any such solution; just need to make sure we choose to do the right thing.

Cc @swamibluedata

[swamibluedata]

We can also make "250m" dynamic based on requests/limit at role level?

[joel-bluedata]

Mmm good point, we could default to whatever the role is requesting.

If we do that, I wonder if we do need to make it customizable? I.e. are there any realistic situations where you actually WOULD want the init container to get more resources than the app container?

[swamibluedata]

If we do that, I wonder if we do need to make it customizable? I.e. are there any realistic situations where you actually WOULD want the init container to get more resources than the app container?

The point you made about copying large amount of data. So yes makes sense to be configurable, which can say

"initcontainer resources should use role Resources or MUST use globally defined Resources"?

[joel-bluedata]

You know... if the pod is going to always claim the max resources from among the init/app requests, then if someone wants more init resources they might as well go ahead and request that larger amount for the app as well. There doesn't seem to be any benefit from having large init resources and then smaller app resources.

I'm kind of inclined just to make BOTH the init resources and the app resources be set to the amount requested for the role, and leave it at that.