Pentaho

 View Only

 Database Repository (Azure): Keep connection open

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Gert Wieland's profile image
Gert Wieland posted 09-16-2019 20:33

I recently switched from a file repository to a database repository (using the cheapest Azure SQL database (S0)). All works fine, but after a few minutes, the connection seems to close, preventing me from saving whatever I was working on. (I can still export to XML and later import, but that's of course not the idea).

 

I get errors like "Couldn't fetch database connections. Invalid state, the Connection object is closed." and "Call collectDatabases() first".

 

Could someone help me out how to keep the connection open?


#Kettle
#Pentaho
#PentahoDataIntegrationPDI
David da Guia Carvalho's profile image
David da Guia Carvalho

Hi,

 

As a cloud service my guess is that you are bound to have some connection issues. If you are on windows you could try:

 

To avoid dropping idle connections by a network component, the following registry settings (or their non-Windows equivalents) should be set on the operating system where the driver is loaded:Registry Setting Recommended ValueHKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ Tcpip \ Parameters \ KeepAliveTime 30000HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ Tcpip \ Parameters \ KeepAliveInterval 1000HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ Tcpip \ Parameters \ TcpMaxDataRetransmissions 10

https://docs.microsoft.com/en-us/sql/connect/jdbc/connecting-to-an-azure-sql-database?view=sql-server-2017

 

But if you are on linux... its gonna be tricker! You could try to put:

KeepAliveTime, KeepAliveInterval, TcpMaxDataRetransmissions into your connection options... but dont get your hopes high!

 

Gert Wieland's profile image
Gert Wieland

Hi David,

 

Thank you very much, that's great information. I'm running a Windows VM on Azure, and I just made the registry changes. I'll post an update later once I worked for a while.

 

But I'd say this is a very promising approach :)

 

Much appreciated,

 

Gert

Gert Wieland's profile image
Gert Wieland

Unfortunately, I'm still getting the same errors :( I'll try switching to a local database (i. e. local on the VM).

(By the way, I'm running CE 8.3)

 

But nonetheless, thanks again for your time and the input!

David da Guia Carvalho's profile image
David da Guia Carvalho

Just a opnion!

Overall, Im not much fan of PDI DB repo. In the past a got a bad experience with it where as the repo grows it was becoming slower (to the point where it was impossible).

Ok it was far down the 3.x and 4.x and from that time to these days a have been using file system repo for most of situations.

The best thing from a fs repo is that you can use any version control to deal with your maps (ktr,kjb) and can apply a CI/CD in a farly simple way.

Also, if you really whats a DB repo (if you intent to go carte, db is probably a better choice) you could try a postgresql or a mysql.

Gert Wieland's profile image
Gert Wieland

Last update, just in case someone comes across this post. I switched to a local MariaDB repository (WampServer), and so far, this seems to run just fine. Probably it wasn't such a good idea to use a cloud database (Azure) as a repository.