Steffen Mueller

Dynamic Carte cluster setup in Nomad/K8s

Discussion created by Steffen Mueller on Jun 12, 2019

Hi all,

I would like to setup a dynamic Carte cluster in a container scheduler system. For this, I managed to create my own Docker image similar to the one described in Diethard Steiner On Business Intelligence. I am also able to start a cluster with a master and one slave server. Everything is working fine. At http://<MASTER_IP>/kettle/getSlaves/ I also get a proper response:

<SlaveServerDetections>
   <SlaveServerDetection>
      <slaveserver>
         <name>Dynamic slave [0.0.0.0:8080]</name>
         <hostname>0.0.0.0</hostname>
         <port>8080</port>
         <webAppName/>
         <username>cluster</username>
         <password>XXX</password>
         <proxy_hostname/><proxy_port/><non_proxy_hosts/>
         <master>N</master>
         <sslMode>N</sslMode>
      </slaveserver>
      <active>Y</active>
      <last_active_date>2019/06/12 17:32:35.600</last_active_date>
      <last_inactive_date/>
   </SlaveServerDetection>
</SlaveServerDetections>

However, I cannot scale up the cluster. When I add another slave, this does not appear. I assume this happens, because the name is not unique. The second, third, and all other nodes will also bind to 0.0.0.0 and port 8080 in the Docker container (to the outside scheduled to other ports by the container scheduler) and the slave name is automatically replaced with these values (-> all instances do have the same name "Dynamic slave [0.0.0.0:8080]") although I provide specific names to the slaves.

Does somebody have a hint for me how to fix this behavior in Carte?

 

Best,

Steffen

Outcomes