Thursday, April 3, 2014

Extracting every nth row

Extracting every nth row

Scenario: How to load every nth row from a Flat file/ relational DB to the target? Suppose n=3, then in above condition the row numbered 3,6,9,12,....so on, This example takes every 3 row to target table.
Solution:
  1. Connect an expression transformation after source qualifier.
    Add the next value port of sequence generator to expression transformation.
    Q14_src_seq_exp_1.PNG
  2. In expression create a new port (validate) and write the expression like in the picture below.
    exp mapping
  3. Connect a filter transformation to expression and write the condition in property like in the picture below.
    filter mapping
  4. Finally connect to target.
    final
 

No comments:

Post a Comment

Data engineering Interview Questions

1)  What all challenges you have faced and how did you overcome from it? Ans:- Challenges Faced and Overcome As a hypothetical Spark develop...