Let us look at how to add a custom header, footer to the file. This is often required for file processing and validation.
|If you would like to Enrich your career with a Talend certified professional, then visit Mindmajix - A Global online training platform: “Talend Certification Course”. This course will help you to achieve excellence in this domain.|
Requirement: Create a delimited file with the below structure.
File Name: Name of the actual File. PID : process ID. Header Columns.
File Created Date: Date & Time. Number of Records: record count.
This is our final job design.
Final Job Design
Step 1: We`ll create Header part first.
Step 2: Add tFixedFlowInput component and configure as below.
"File Name: CustomerDetails.csv" "PID :"+pid "Id,Name,City,State,Street"
Step 3: Add tFileOutputDelimited and connect with tFixedFlowInput component using Main flow, then configured as follows.
Step 4: Add tRowGenerator
Step 5: Copy Paste tFileOutputDelimited_1 component which was created in Step 3, then connect tFileOutputDelimited_2 with tRowGenerator using main flow. Configure as follows.
Step 6: Create a context variable named as “NumberOfRows”, then copy and paste tFixedFlowInput_1.
Step 7: Configure newly pasted tFixedFlowInput_2 component.
Step 8: Copy paste tFileOutputDelimited_2 and connect with tFixedFlowInput_2 using main flow. No need to do any additional configuration.
Step 9: Add tFlowMeterCatcher component and add java row and connect each other using main flow, don`t click on “Synch” button on tJavaRow. Write below code in tJavaRow component.
Step 10: Run the job, it will create files with the Header part, Data part, and footer part. Below is our final output.
Header and Footer Output
|Related Article: Talend Interview Questions and Answers|
This recipe shows how to parse a file that has a header and trailer records, and a record type at the start of a line.
Open the jo_cook_ch08_0060_headTrailtMap job.
How to accomplish it…
The steps for reading headers and trailers using tMap are as follows:
sc_cook_ch8_0060_genericCustomerHeader sc_cook_ch8_0060_genericCustomerDetail sc_cook_ch8_0060_genericCustomerTrailer
|Related Article: Talend Tutorial for Beginners|
How it works…
tFileInputFullRow allows us to read a row of any format into tMap. This is important because we do not want records to be rejected due to schema errors at this stage. The start of each row is then tested for the record type; 00, 01, or 02, the header, detail, or trailer records respectively. The different rows are then passed to a tExtractDelimitedFields component for breaking down into the individual schema columns.
This isn’t the only method of reading files with headers and trailers, and in fact, the best Talend method would be to use the tFileInputMSDelimited component, for this example. This method, however, is much more flexible, in which the conditions for sending in the data as an output to each of the flows do not depend upon a fixed field being present.
This recipe shows how to parse a file that has a header and trailer records but does not have an associated record type. Instead, the header is the first record in the file, and the trailer is the last record in the file.
Open the jo_cook_ch08_0070_headTrailtMapNoType job. You will see that it is a slightly changed version of the completed job from the previous recipe; the output schemas have changed.
How to achieve it…
The steps for reading headers and trailers with no identifiers are as follows:
which is the same as our input file.
Connect an onSubJobOk trigger from the tFileRowCount component to the tFileInputDelimited.
Open the tMap, and add a new variable rowCount. Set its expression to Numeric.sequence(“rowNumber”,1,1).
Change the Filter expressions for header, detail, and trailer to those shown as follows:
Var.rowNumber == 1 Var.rowNumber != ((Integer)globalMap.get("tFileRowCount_1_COUNT")) Var.rowNumber == ((Integer)globalMap.get("tFileRowCount_1_COUNT"))
|Explore TALEND Sample Resumes! Download & Edit, Get Noticed by Top Employers!|
How it works…
The tFileRowCount component tells us how many rows are present in the file.
In the tMap, we use a sequence to calculate the current line number. If the line number is 1, then we have a header row. If it is equal to the row count (held in globalMap), then we have a trailer row, and all other rows are detail rows.
We then use the tExtractDelimitedFields to extract the individual delimited fields into a different schema for each of the row types.
Stay updated with our newsletter, packed with Tutorials, Interview Questions, How-to's, Tips & Tricks, Latest Trends & Updates, and more ➤ Straight to your inbox!
|Talend Training||Jun 03 to Jun 18|
|Talend Training||Jun 06 to Jun 21|
|Talend Training||Jun 10 to Jun 25|
|Talend Training||Jun 13 to Jun 28|
Ravindra Savaram is a Content Lead at Mindmajix.com. His passion lies in writing articles on the most popular IT platforms including Machine learning, DevOps, Data Science, Artificial Intelligence, RPA, Deep Learning, and so on. You can stay up to date on all these technologies by following him on LinkedIn and Twitter.
Copyright © 2013 - 2023 MindMajix Technologies