Pentaho

 View Only
Back to discussions
Expand all | Collapse all

s3 file output Step connect timed out us-west-2

This thread has been viewed 72 times

  • 1.  s3 file output Step connect timed out us-west-2

    Posted 05-13-2022 04:25

    I am using Pentaho 8.3 and running the job through a shell script on an Linux EC2 Dev server.

    Does the s3 file output step need the aws credentials set in the profile or in the .aws/config please?

    Does Pentaho need the full credentials set in the .aws/credentials please?

    I am getting this error:

    S3 file output.0 - Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to dat-xxx-dev-logs.s3.us-west-2.amazonaws.com:443 [dat-xxx-dev-logs.s3.us-west-2.amazonaws.com/52.92.195.58] failed: connect timed out

    Regards Vince

     



  • 2.  RE: s3 file output Step connect timed out us-west-2

    Posted 05-16-2022 03:56
    This new thread carries on from the old question, asked earlier in the old thread.
    This new thread, with the same name, has been set up because the old thread is broken.
    Please reply to this new thread and change to a discussion.

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


  • 3.  RE: s3 file output Step connect timed out us-west-2

    Posted 05-17-2022 19:52

    Hi Vince

    Try adding

    echo "currentuser=$USER  " > /tmp/awscheck.log
    echo "homedir=$HOME" >> /tmp/awscheck.log
    echo ` aws configure list` >> /tmp/awscheck.log

    to the shell script and then check the output in /tmp/awscheck.log    Just to confirm that all the environment variables have been set on the EC2 side as you expect (bash can be tricky running scripts remotely)

    Alternatively, try adding the appropriate variable at the head of teh script

    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    export AWS_DEFAULT_REGION=eu-west-2



    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------

    Original Message


  • 4.  RE: s3 file output Step connect timed out us-west-2

    Posted 05-18-2022 09:46
    Thank you for the Reply Andrew,

         Our platform support team here will not allow me to set the: AWS_ACCESS_KEY_ID or theAWS_SECRET_ACCESS_KEY but I have set AWS_DEFAULT_REGION=euwest-2

        The new echo statements reveal that the profile is not set but the region is:

    + echo currentuser=ics2
    + echo homedir=/home/ics2
    ++ aws configure list
    + echo Name Value Type Location ---- ----- ---- -------- profile '<not' 'set>' None None access_key '****************XYAV' iam-role secret_key '****************qw2g' iam-role region eu-west-2 config-file '~/.aws/config'

    Question:     Do you know how to force the Pentaho Step to use the PDI EC2 user Instance profile and not AWS Profile please?

                           Our Platform Support Team will not allow AWS Credentials to be used!

                            We are trying to use this supporting documentation:

    AWS credentials

    The S3 File Output step provides credentials to the Amazon Web Services SDK for Java using a credential provider chain. The default credential provider chain looks for AWS credentials (we use option 5):

    • Instance profile credentials

      These credentials are delivered through the Amazon EC2 metadata service, and can be used on EC2 instances with an assigned instance role.

    The S3 File Output step can use any of these methods to authenticate AWS credentials. For more information on setting up AWS credentials, see Working with AWS Credentials.



    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


  • 5.  RE: s3 file output Step connect timed out us-west-2

    Posted 06-02-2022 20:44
      |   view attached
    hi Vince

    I've attached a demo using a shell script step instead that might get around your authentication issue.

    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------

    Attachment(s)

    ktr
    archive_files_to_s3.ktr   24 KB 1 version
    Original Message


  • 6.  RE: s3 file output Step connect timed out us-west-2

    Posted 06-21-2022 13:00

    Andrew,

                    Thank You so much for your Reply. I have been on Holiday and missed this update after being given other work to do. I have used similar Pentaho s3 steps to run aws s3 mv and aws s3 copy Utility, statements but we do not want to use batch mode.

                    The requirement here is for real time logging, in fact.

                    A work-around could be, writing to a temporary Table and using the aws s3 copy utility but the requirement is real time not batch.

                    We want to use option 5 from the supporting Documentation:

    AWS CREDENTIALS

    The S3 File Output step provides credentials to the Amazon Web Services SDK for Java using a credential provider chain. The default credential provider chain looks for AWS credentials:

    • Instance profile credentials

    These credentials are delivered through the Amazon EC2 metadata service, and can be used on EC2 instances with an assigned instance role.



    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


  • 7.  RE: s3 file output Step connect timed out us-west-2

    Posted 06-22-2022 16:29

    Vince,

    You might be struggling will trying to fall through the credential providers.  I seem to remember having an old creditials file and it was stopping me even with IAM roles in place.   Not on AWS anymore to test and my memory may be faulty.

    I know that I have been able to add an S3 role to an EC2 instance and then access via the CLI as Andrew showed. You might want to make sure that there are no variables set and no credentials files in place that might be sending you down the wrong authentication pathway. 

    Could also turn on debug logging for  com.amazonaws.auth.AWSCredentialsProviderChain



    ------------------------------
    Stephen Donovan
    Digital Solutions Architect
    Hitachi Vantara
    ------------------------------

    Original Message


  • 8.  RE: s3 file output Step connect timed out us-west-2

    Posted 07-30-2022 15:17
    Vince
    Wondering if you are still experiencing this timeout? Can you share an update to see if I need to look into this?

    ------------------------------
    Carlos Lopez
    Application Architecture Engineering - Expert
    Hitachi Vantara
    ------------------------------

    Original Message


  • 9.  RE: s3 file output Step connect timed out us-west-2

    Posted 08-01-2022 03:33
    Carlos,
         Thank you for the reply but there is no change.

        The Instance profile credentials, do not appear to work with an assigned instance role for the S3 File Output step.
    Regards Vince

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


  • 10.  RE: s3 file output Step connect timed out us-west-2

    Posted 08-19-2022 11:53
    We have a Service Patch that has not been applied correctly and this is thought to be the cause of the error.
    The Service Patch will have to be re-applied and we will start again.
    Thank you for you help and interest, so far.

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


Related Content