Request More Information

We are hiring!

Xtivia, Inc.
2013 North America Liferay Partner of the Year

Join our top-notch team of Enterprise Java and Liferay Portal professionals in our Austin, TX office or work remotely! Please check out our current Dice listings.

Recent Bloggers

Luke A Smith
Posts: 17
Stars: 0
Date: 10/20/14
Sonu Verghese
Posts: 7
Stars: 0
Date: 10/16/14
Chris Shaw
Posts: 8
Stars: 0
Date: 10/15/14
Sushil Shirodkar
Posts: 1
Stars: 0
Date: 8/28/14
Vamshidhar Srikantapuram
Posts: 11
Stars: 0
Date: 8/8/14
Barrie Shaw
Posts: 6
Stars: 0
Date: 8/4/14
Omar Yimaier
Posts: 4
Stars: 0
Date: 8/1/14
Bob Dietrich
Posts: 8
Stars: 0
Date: 8/1/14
Matt Wolinski
Posts: 2
Stars: 0
Date: 7/30/14
Richard Ngo
Posts: 1
Stars: 0
Date: 6/19/14
« Back

Looping Construct for ETL - Simplified by Talend DI

I am not a regular ETL developer but I do "dabble" in developing code from time to time. One requirement that I come across is looping (while, for, infinite) in a job and performing a task based on condition and then going to sleep for a period of time. This certainly can be accomplished through a scheduler as well but is often time easier to accomplish in the ETL job design.

I have had the opportunity to implement this construct using SSIS and Datastage. In my opinion, Datastage was the most complicated - took me a good 5 hours to get the code to do exactly what I wanted. SSIS was more cooperating - I was able to do the same in about 3 hours.

As I started working with Talend Open Studio earlier this week, I was amazed to see how simple it was to work with Talend.

As you open the Studio, you can find various orchestration steps very quickly - something the other two tools do not do. i.e. the IDE user interface is more intuitive for Talend

From here, it was drag and drop easy...

Step 1: Drag the while loop on the canvas - define the parameters.

 

Step 2: Create a job run step to be executed in each of the iterations

Step 3: Drag the "sleep" step and assign values.

I was done start to finish in about 20 minutes and the code generated in Java was pretty easy to read and understand. Very cool!

 

Comments
Trackback URL:

Join the discussion! Speak your mind! Be the first.

Request More Information