I am trying to go through mutilple pages to get all the data, currently the rest call only lists 1000 entries per page a nd there are 27 pages, how to loop through all pages?
This doesn't sound like a Pentaho question, this is a question for how to use the API in particular you're working with. You'll need to figure out how to manipulate the API such that it will provide all ~27000 entries with one call, or you'll need to figure out how to get the entries from separate pages. I'd suggest you check if the output with the same REST command is different or the same in cURL versus the Pentaho REST client.
If you believe I've misunderstood your issue, please clarify with a more detailed description of the issue you're facing.
Retrieving data ...