Pentaho

 View Only
  • 1.  s3 file output Step connect timed out us-west-2

    Posted 15 days ago

    I am using Pentaho 8.3 and running the job through a shell script on an Linux EC2 Dev server.

    Does the s3 file output step need the aws credentials set in the profile or in the .aws/config please?

    Does Pentaho need the full credentials set in the .aws/credentials please?

    I am getting this error:

    S3 file output.0 - Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to dat-xxx-dev-logs.s3.us-west-2.amazonaws.com:443 [dat-xxx-dev-logs.s3.us-west-2.amazonaws.com/52.92.195.58] failed: connect timed out

    Regards Vince

     



  • 2.  RE: s3 file output Step connect timed out us-west-2

    Posted 12 days ago
    This new thread carries on from the old question, asked earlier in the old thread.
    This new thread, with the same name, has been set up because the old thread is broken.
    Please reply to this new thread and change to a discussion.

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------



  • 3.  RE: s3 file output Step connect timed out us-west-2

    Posted 11 days ago

    Hi Vince

    Try adding

    echo "currentuser=$USER  " > /tmp/awscheck.log
    echo "homedir=$HOME" >> /tmp/awscheck.log
    echo ` aws configure list` >> /tmp/awscheck.log

    to the shell script and then check the output in /tmp/awscheck.log    Just to confirm that all the environment variables have been set on the EC2 side as you expect (bash can be tricky running scripts remotely)

    Alternatively, try adding the appropriate variable at the head of teh script

    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    export AWS_DEFAULT_REGION=eu-west-2



    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------



  • 4.  RE: s3 file output Step connect timed out us-west-2

    Posted 10 days ago
    Thank you for the Reply Andrew,

         Our platform support team here will not allow me to set the: AWS_ACCESS_KEY_ID or theAWS_SECRET_ACCESS_KEY but I have set AWS_DEFAULT_REGION=euwest-2

        The new echo statements reveal that the profile is not set but the region is:

    + echo currentuser=ics2
    + echo homedir=/home/ics2
    ++ aws configure list
    + echo Name Value Type Location ---- ----- ---- -------- profile '<not' 'set>' None None access_key '****************XYAV' iam-role secret_key '****************qw2g' iam-role region eu-west-2 config-file '~/.aws/config'

    Question:     Do you know how to force the Pentaho Step to use the PDI EC2 user Instance profile and not AWS Profile please?

                           Our Platform Support Team will not allow AWS Credentials to be used!

                            We are trying to use this supporting documentation:

    AWS credentials

    The S3 File Output step provides credentials to the Amazon Web Services SDK for Java using a credential provider chain. The default credential provider chain looks for AWS credentials (we use option 5):

    • Instance profile credentials

      These credentials are delivered through the Amazon EC2 metadata service, and can be used on EC2 instances with an assigned instance role.

    The S3 File Output step can use any of these methods to authenticate AWS credentials. For more information on setting up AWS credentials, see Working with AWS Credentials.



    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------