Using extraPortMappings in Windows

[farzadmf]

I'm using Docker Desktop on Windows and I'm trying to figure out how to use extraPortMappings to access my services

I'm trying with 2 clusters I created using kind, the first one:

kind create cluster --config kind-config.yaml

And my config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4

nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 80
    hostPort: 8080

And I do these:

kubectl create deploy mygninx --image=nginx
kubectl expose deploy mynginx --type NodePort --port 80

Then, when I try to access:

curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused

So, I created another cluster to test:

kind create cluster --name kind2 --config kind-config.yaml

And my config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4

nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 80
    hostPort: 8081
- role: worker

And I try to access this one:

curl localhost:8081
curl: (52) Empty reply from server

I tried to search in the issues, but I couldn't find a similar one, so sorry if this is a duplicate issue

[BenTheElder]

kubernetes nodeports are in the range 30000-32767 by default. when you request 80 you get a random port in that range instead.

https://kubernetes.io/docs/concepts/services-networking/service/#nodeport

if you forward one of those ports it should work, otherwise you'll want to consider e.g. an ingress, or using a hostport https://kind.sigs.k8s.io/docs/user/ingress/

on linux you can use type=loadblancer using metallb, but not on windows currently due to docker's networking model
https://kind.sigs.k8s.io/docs/user/resources/#how-to-use-kind-with-metalllb

[farzadmf]

Thank you @BenTheElder for your answer, a few questions:

• I thought I am using the hostPort, aren't I? In the config, I have hostPort: 8080, so I thought that means that the port 8080 would be the mapped port I can use

• When you say "when you request 80", you mean when I run kubectl expose command? If so, as you mentioned, if I don't configure the nodePort in a service YAML file (I don't think it's possible through kubectl expose command), I would get a random port, so I don't think I can forward it in advance, right?

• Is it somehow possible to map ports "on-the-fly" (after I create my cluster without requiring to re-create the cluster)? Or everything should be known before creating the cluster?

Thank you for the tip regarding metallb; I'll definitely check that out on my home Linux machine. However, I would really like to be able to try things in my work Windows machine too (I think I can live without the load balancer, but I need to be able to call my services)

[BenTheElder]

Quick example (not tested on windows).

I tend to use yaml and kubectl apply -f for anything more interesting and had forgotten that kubectl expose ... doesn't support setting the nodeport. however kubectl create service nodeport ... does (xref: https://github.com/kubernetes/kubernetes/issues/25478)|

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - hostPort: 31111
    containerPort: 31111

kubectl create deploy mynginx --image=nginx
kubectl create service nodeport --node-port=31111 mynginx --tcp=80:80

[BenTheElder]

I thought I am using the hostPort, aren't I? In the config, I have hostPort: 8080, so I thought that means that the port 8080 would be the mapped port I can use

Er sorry, I mean in the kubernetes objects (your pods) rather than in kind.
You also need the kind hostport, kind closely follows Kubernetes container runtime interface naming and types in many places.

When you say "when you request 80", you mean when I run kubectl expose command? If so, as you mentioned, if I don't configure the nodePort in a service YAML file (I don't think it's possible through kubectl expose command), I would get a random port, so I don't think I can forward it in advance, right?

Before your comment showed up I wrote #1523 (comment), you can see how to do it here without yaml (use kubectl create service nodeport instead of kubectl expose deployment)

Is it somehow possible to map ports "on-the-fly" (after I create my cluster without requiring to re-create the cluster)? Or everything should be known before creating the cluster?

It's not possible natively in kind (or docker), but you can run a "sidecar" container next to the node that just proxies traffic (something like socat) from some port mapped to the host to some port in the node.

Thank you for the tip regarding metallb; I'll definitely check that out on my home Linux machine. However, I would really like to be able to try things in my work Windows machine too (I think I can live without the load balancer, but I need to be able to call my services)

Yeah, I would really like this to work, but docker containers are not reachable by IP from the host on windows / mac which is pretty limiting, we have to use these magicalish port forwards to be portable.

We have some thoughts about workarounds but ... they're very involved and unlikely to be ready anytime soon, if ever.

[BenTheElder]

To elaborate a bit...

• You do not have to use the same ports all the way down, but I find sometimes it's less confusing, you don't need to worry too much about the layers if it's ~all the same port :-)

• You don't have to use hostPort in your Kubernetes objects, but it's a super simple approach if you control the pods and stick to one node. For anything more complex you probably want a nodePort, ingress, or loadbalancer (if you're on linux)

• kind generally favors teardown / re-create (for clean testing state) and has focused on making that work well and fast(er) so far. very recently we've put more effort into e.g. Cluster doesn't restart when docker restarts #148 and may revisit this in the future, but broadly right now you cannot modify the cluster spec without recreating. Even if we did support this, docker doesn't support dynamically adding port forwards. You can try the socat type workaround though.

[farzadmf]

Thank you @BenTheElder for your SUPER fast replies 🙂, I appreciate it

I will try things on my Windows machine tomorrow and get back to this issue.

Also, sorry if I'm asking a stupid question, but would you by any chance be able to give some pointers about socat you mentioned; I hadn't heard about it before (of course, I don't expect you to, but if you do, it would be very much appreciated)

And, I'm OK with trying things to hopefully make kind work on my Windows machine: I was so excited a couple of nights ago when I re-discovered kind and its convenience to be able to create even multi-node clusters (the limitation I was unhappy with minikube)

PS: I know Windows makes things much harder, but unfortunately I don't have a choice regarding my work machine. Also, since Docker Desktop on Windows requires Hyper-V, I'm having all sorts of problems with VirtualBox, and I can't test things there. All this to say that I regard kind as my saviour here 😝

[BenTheElder]

Thank you @BenTheElder for your SUPER fast replies slightly_smiling_face, I appreciate it

I appreciate the appreciation :-)

Also, sorry if I'm asking a stupid question, but would you by any chance be able to give some pointers about socat you mentioned; I hadn't heard about it before (of course, I don't expect you to, but if you do, it would be very much appreciated)

Sure, there's a pretty good example here under "Use Case: Publish a port on an existing container":
https://hub.docker.com/r/alpine/socat/

Please do not use --link though, this is deprecated in docker , we can instead tweak this slightly...

For the currently released version of kind you can do:

1) get the node IP:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' kind-control-plane

or

kubectl get nodes -ojsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'

For my cluster right now the node IP is 172.17.0.3
You can see this as the internal IP in kubectl get no -owide

$ kubectl get no -owide
NAME                 STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
kind-control-plane   Ready    master   46m   v1.18.2   172.17.0.3    <none>        Ubuntu 19.10   5.4.0-26-generic   containerd://1.3.3-14-g449e9269

2) Create a container forwarding a port to another port on this IP

docker run --publish 4321:1234 alpine/socat tcp-listen:1234,fork,reuseaddr tcp-connect:172.17.0.3:1234

Ccurrently in the kind version under development you will need to pass --net=kind to this docker run as well, we've started leveraging a user defined network to fix various things. You'll do this instead:

1) Create a container forwarding a port to another port on this IP

docker run --publish 4321:1234 alpine/socat tcp-listen:1234,fork,reuseaddr tcp-connect:kind-control-plane:1234

PS: I know Windows makes things much harder, but unfortunately I don't have a choice regarding my work machine. Also, since Docker Desktop on Windows requires Hyper-V, I'm having all sorts of problems with VirtualBox, and I can't test things there. All this to say that I regard kind as my saviour here stuck_out_tongue_closed_eyes

I totally sympathize, this problem applies to mac as well :-)
I haven't had a chance to explore the WSL2 backend more, this may improve things.

[BenTheElder]

This needs to become a guide in the docs once we're done mucking with the network details, almost there, I hope :-)

[farzadmf]

Thank you again @BenTheElder for your detailed instructions.

Just to make sure I'm not just confusing you with my machine, it may be worth noting that this is not my personal laptop (and it's running Windows), so is it possible that something is messing things up in the network causing the behavior I see below?

For example, initially, I had a HUGE delay when I was running Docker commands, and for some reason, adding the following line in my hosts file resolved that issue:

127.0.0.1 <my-company-name>

so, I was wondering if it would be possible for this line to mess things up!

---

I'm not sure if I'm missing something, but scoat doesn't seem to work for me (I'm testing on Windows right now). This is what I'm doing:

1. My kind config only contains two nodes (without extraPortMappings because I think it's not required with socat, I hope that's true):

kind create cluster --config C:\bin\temp\remain\kind-config.yaml

2. I create my deployment and service like this:

kubectl create deploy mygninx --image=nginx
kubectl create service nodeport mynginx --node-port=32000 --tcp=80:80

My service: kubectl get svc mynginx -o wide:

NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE   SELECTOR
mynginx   NodePort   10.104.203.251   <none>        80:32000/TCP   25s   app=mynginx

The output of kubectl get nodes -o wide:

NAME                 STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
kind-control-plane   Ready    master   10m     v1.18.2   172.18.0.3    <none>        Ubuntu 19.10   4.19.76-linuxkit   containerd://1.3.3-14-g449e9269
kind-worker          Ready    <none>   9m59s   v1.18.2   172.18.0.2    <none>        Ubuntu 19.10   4.19.76-linuxkit   containerd://1.3.3-14-g449e9269

3. Then, I'm running socat:

1. Create a container forwarding a port to another port on this IP
   docker run --publish 4321:1234 alpine/socat tcp-listen:1234,fork,reuseaddr tcp-connect:kind-control-plane:1234

I think here, you forgot the --net=kind that you mentioned. So, according to your recommendation, I run this:

docker run --name mysocat --network kind --rm --publish 4321:1234 alpine/socat tcp-
listen:1234,fork,reuseaddr tcp-connect:kind-control-plane:32000

However, this is what I get:

curl localhost:4321
curl: (52) Empty reply from server

And, in my terminal running socat, I see this log:

2020/04/28 16:15:24 socat[6] E connect(5, AF=2 172.18.0.3:32000, 16): Connection refused

I also tried forwarding to kind-control-plane:80 in my socat, but I get the same result.

Sorry if this got too long 😛

[BenTheElder]

For example, initially, I had a HUGE delay when I was running Docker commands, and for some reason, adding the following line in my hosts file resolved that issue:

not sure about that

My kind config only contains two nodes (without extraPortMappings because I think it's not required with socat, I hope that's true):

yes, though I recommend you only use one node unless you have a good reason for more.
unlike physical nodes you don't get much isolation or more resources and it complicates things.

I think here, you forgot the --net=kind that you mentioned. So, according to your recommendation, I run this:

er right, --net=kind only applies to kind installed from HEAD / the latest code. v0.7.0 does not have this. only this version can use the hostname as well.

if you're using kind v0.7.0 or any build not extremely recent you need to use the instructions involving getting the IP.

[farzadmf]

--net=kind only applies to kind installed from HEAD / the latest code

The version I have is this: kind v0.8.0-alpha go1.14.2 windows/amd64, does it support the hostname?

[farzadmf]

Hey @BenTheElder , so I think I've made some progress; the socat solution seems to be working when I create a cluster with a single node 🙂

I recommend you only use one node unless you have a good reason for more
I'm wondering if this is a limitation (or recommendation) only for non-native Docker environments (basically, non-linux), or it applies to all environments

One thing that was super exciting for me was the fact that I can have multi-node clusters with kind I'm doing that now in my Linux environment and using LXD, but I was so excited to see that I can have that in a Windows environment too. Also, I thought that "that's so cool, now it won't matter where I am (Linux or Windows), I can use the same tool and get the same results"; I guess that's not 100% true, is it?

[BenTheElder]

sorry got a bit swamped ...

One thing that was super exciting for me was the fact that I can have multi-node clusters with kind I'm doing that now in my Linux environment and using LXD, but I was so excited to see that I can have that in a Windows environment too. Also, I thought that "that's so cool, now it won't matter where I am (Linux or Windows), I can use the same tool and get the same results"; I guess that's not 100% true, is it?

Yeah...

Multi-Node clusters should work everywhere, but do add some more confusion to things like network mapping :-)

The most noticeable difference on Linux vs Not-Linux wrt Docker is the network connectivity between the host and the containers, on linux you can poke each container by IP and do all kinds of fun stuff. On mac / windows you'll have to use port forwards where you might not have needed them and generally work around the minimal connectivity :(

We strive to make everything we officially ship work the same everywhere as much as possible, and it pretty much does. It does restrict us from e.g. shipping a kubernetes LoadBalancer implementation currently though.

[BenTheElder]

The version I have is this: kind v0.8.0-alpha go1.14.2 windows/amd64, does it support the hostname?

it probably does, that build unfortunately doesn't have the commit (make does, but not plain go build ...)

v0.8.0 actual is out now 😅

[farzadmf]

sorry got a bit swamped ...
Don't worry about it at all, I really appreciate you putting the time to answer my questions here

v0.8.0 actual is out now 😅
Given that I was using the alpha version, I don't think there are those many new changes for me?

One thing confusing for me is why it works with a single node and not multi-node (in case of socat). In theory, the socat container should be my "ingress" here, right?

[BenTheElder]

sorry I lost this message.

Your socat thing needs to forward to where the clsuter is listening, in the multi-node case you'll either want to ensure the workload is "pinned" to a node via label + label selector, or use a nodePort (which is on all nodes).

[farzadmf]

Thank you @BenTheElder for your answer

[afreisinger]

Thank you again @BenTheElder for your detailed instructions.

Just to make sure I'm not just confusing you with my machine, it may be worth noting that this is not my personal laptop (and it's running Windows), so is it possible that something is messing things up in the network causing the behavior I see below?

For example, initially, I had a HUGE delay when I was running Docker commands, and for some reason, adding the following line in my hosts file resolved that issue:

127.0.0.1 <my-company-name>

so, I was wondering if it would be possible for this line to mess things up!

I'm not sure if I'm missing something, but scoat doesn't seem to work for me (I'm testing on Windows right now). This is what I'm doing:

1. My kind config only contains two nodes (without extraPortMappings because I think it's not required with socat, I hope that's true):

kind create cluster --config C:\bin\temp\remain\kind-config.yaml

1. I create my deployment and service like this:

kubectl create deploy mygninx --image=nginx
kubectl create service nodeport mynginx --node-port=32000 --tcp=80:80

My service: kubectl get svc mynginx -o wide:

NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE   SELECTOR
mynginx   NodePort   10.104.203.251   <none>        80:32000/TCP   25s   app=mynginx

The output of kubectl get nodes -o wide:

NAME                 STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
kind-control-plane   Ready    master   10m     v1.18.2   172.18.0.3    <none>        Ubuntu 19.10   4.19.76-linuxkit   containerd://1.3.3-14-g449e9269
kind-worker          Ready    <none>   9m59s   v1.18.2   172.18.0.2    <none>        Ubuntu 19.10   4.19.76-linuxkit   containerd://1.3.3-14-g449e9269

1. Then, I'm running socat:

1. Create a container forwarding a port to another port on this IP
   docker run --publish 4321:1234 alpine/socat tcp-listen:1234,fork,reuseaddr tcp-connect:kind-control-plane:1234

I think here, you forgot the --net=kind that you mentioned. So, according to your recommendation, I run this:

docker run --name mysocat --network kind --rm --publish 4321:1234 alpine/socat tcp-
listen:1234,fork,reuseaddr tcp-connect:kind-control-plane:32000

However, this is what I get:

curl localhost:4321
curl: (52) Empty reply from server

And, in my terminal running socat, I see this log:

2020/04/28 16:15:24 socat[6] E connect(5, AF=2 172.18.0.3:32000, 16): Connection refused

I also tried forwarding to kind-control-plane:80 in my socat, but I get the same result.

Sorry if this got too long 😛

Hi @farzadmf

How do you solve the issue about Connection refused on socat terminal ?

[farzadmf]

Hi @farzadmf

How do you solve the issue about Connection refused on socat terminal ?

Hey @afreisinger , it's been a while on this issue, 🙂. TBH, I'm not sure if I was even able to solve the issue!

I'll try to remember and let you know if something comes to mind.