Pentaho

 View Only

 s3 file output Step connect timed out us-west-2

Vince Popplewell's profile image
Vince Popplewell posted 05-10-2022 12:02

The Pentaho Step S3 file output is picking up the Region = us-west-2 by default

How do we force it to look at the config and not default to Region = us-west-2 please? 

We are using profile instance

The environment .aws/config is set to region = eu-west-2

Andrew Cave's profile image
Andrew Cave
Hi Vince

So in the file it reads

~/.aws/config

[default] region=eu-west-2
and that file is in the home directory of the account that is running the pentaho instance?

try running the transformation with Debug logging on and see where Kettle is trying to read the config from.
Vince Popplewell's profile image
Vince Popplewell
Yes that file is in the root directory (I disguised name to pdi-xxx)

[root@pdi-xxx ~]# cd .aws
[root@pdi-xx .aws]# ls
config
[root@pdi-xxx .aws]# more config
[default]
region = eu-west-2
[root@xx .aws]#

I have -level=Debug and i am getting the same time out error: 
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to dat-nonlive-xxx-dev-logs.s3.us-west-2.amazonaws.com:443 [dat-nonlive-xxx-logs.s3.us-west-2.amazonaws.com/52.218.229.1] failed: connect timed out

Does the config need more parameters like access_key or secret_key? Because these have been set up in the Profile.


Stephen Donovan's profile image
Stephen Donovan

I believe you are on the right track.  Kettle simply uses the java SDK from Amazon.  So the authentication options cycle through in order as specified here.

https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

Not as familiar with CE, but double check your VFS connection set up.  The region selection is in that dialog an may be overriding the credentials file.

Attachment  View in library
image.png 45 KB
Vince Popplewell's profile image
Vince Popplewell
Thank you for your reply Stephen,

     What is VFS please?
Stephen Donovan's profile image
Stephen Donovan

VFS is Virtual FileSystem.  It allows you to reference s3, hadoop and others in a generic way in the dialogs.  Enabled in many of the input and output steps.  For example s3:  Would be configured as a VFS connection and then referenced as s3://.  Forgive me, as I have always set it up this way, have to dig deeper to see what other path you might be trying.

VFS configurations are in the same View tab as the database connections.  Right click New...

Vince Popplewell's profile image
Vince Popplewell
Please reply to the new thread used for the discussion.
The old thread is broken!