This article demonstrates how to parse a file with header and trailer records, as well as a record type at the beginning of each line.
Let us look at how to add a custom header, footer to the file. This is often required for file processing and validation.
If you would like to Enrich your career with a Talend certified professional, then visit Mindmajix - A Global online training platform: “Talend Certification Course”. This course will help you to achieve excellence in this domain. |
Requirement: Create a delimited file with the below structure.
Header Part
File Name: Name of the actual File.
PID : process ID.
Header Columns.
Body Part
Text Data.
Footer Part
File Created Date: Date & Time.
Number of Records: record count.
This is our final job design.
Final Job Design
Step 1: We`ll create Header part first.
Step 2: Add tFixedFlowInput component and configure as below.
"File Name: CustomerDetails.csv"
"PID :"+pid
"Id,Name,City,State,Street"
tFixedFlowInput Setting
Step 3: Add tFileOutputDelimited and connect with tFixedFlowInput component using Main flow, then configured as follows.
Step 4: Add tRowGenerator
Step 5: Copy Paste tFileOutputDelimited_1 component which was created in Step 3, then connect tFileOutputDelimited_2 with tRowGenerator using main flow. Configure as follows.
Step 6: Create a context variable named as “NumberOfRows”, then copy and paste tFixedFlowInput_1.
Step 7: Configure newly pasted tFixedFlowInput_2 component.
Step 8: Copy paste tFileOutputDelimited_2 and connect with tFixedFlowInput_2 using main flow. No need to do any additional configuration.
Step 9: Add tFlowMeterCatcher component and add java row and connect each other using main flow, don`t click on “Synch” button on tJavaRow. Write below code in tJavaRow component.
context.NumberOfRows=input_row.label+input_row.count;
Step 10: Run the job, it will create files with the Header part, Data part, and footer part. Below is our final output.
Header and Footer Output
Related Article: Talend Interview Questions and Answers |
This recipe shows how to parse a file that has a header and trailer records, and a record type at the start of a line.
Getting ready
Open the jo_cook_ch08_0060_headTrailtMap job.
How to accomplish it…
The steps for reading headers and trailers using tMap are as follows:
sc_cook_ch8_0060_genericCustomerHeader
sc_cook_ch8_0060_genericCustomerDetail
sc_cook_ch8_0060_genericCustomerTrailer
Related Article: Talend Tutorial for Beginners |
How it works…
tFileInputFullRow allows us to read a row of any format into tMap. This is important because we do not want records to be rejected due to schema errors at this stage. The start of each row is then tested for the record type; 00, 01, or 02, the header, detail, or trailer records respectively. The different rows are then passed to a tExtractDelimitedFields component for breaking down into the individual schema columns.
There’s more…
This isn’t the only method of reading files with headers and trailers, and in fact, the best Talend method would be to use the tFileInputMSDelimited component, for this example. This method, however, is much more flexible, in which the conditions for sending in the data as an output to each of the flows do not depend upon a fixed field being present.
This recipe shows how to parse a file that has a header and trailer records but does not have an associated record type. Instead, the header is the first record in the file, and the trailer is the last record in the file.
Getting ready
Open the jo_cook_ch08_0070_headTrailtMapNoType job. You will see that it is a slightly changed version of the completed job from the previous recipe; the output schemas have changed.
How to achieve it…
The steps for reading headers and trailers with no identifiers are as follows:
context.cookbookData+"/chapter8/chapter08_jo_0070_customerData.txt
which is the same as our input file.
Connect an onSubJobOk trigger from the tFileRowCount component to the tFileInputDelimited.
Open the tMap, and add a new variable rowCount. Set its expression to Numeric.sequence(“rowNumber”,1,1).
Change the Filter expressions for header, detail, and trailer to those shown as follows:
Var.rowNumber == 1
Var.rowNumber !=
((Integer)globalMap.get("tFileRowCount_1_COUNT"))
Var.rowNumber ==
((Integer)globalMap.get("tFileRowCount_1_COUNT"))
Explore TALEND Sample Resumes! Download & Edit, Get Noticed by Top Employers! |
How it works…
The tFileRowCount component tells us how many rows are present in the file.
In the tMap, we use a sequence to calculate the current line number. If the line number is 1, then we have a header row. If it is equal to the row count (held in globalMap), then we have a trailer row, and all other rows are detail rows.
We then use the tExtractDelimitedFields to extract the individual delimited fields into a different schema for each of the row types.
Our work-support plans provide precise options as per your project tasks. Whether you are a newbie or an experienced professional seeking assistance in completing project tasks, we are here with the following plans to meet your custom needs:
Name | Dates | |
---|---|---|
Talend Training | Dec 07 to Dec 22 | View Details |
Talend Training | Dec 10 to Dec 25 | View Details |
Talend Training | Dec 14 to Dec 29 | View Details |
Talend Training | Dec 17 to Jan 01 | View Details |
Ravindra Savaram is a Technical Lead at Mindmajix.com. His passion lies in writing articles on the most popular IT platforms including Machine learning, DevOps, Data Science, Artificial Intelligence, RPA, Deep Learning, and so on. You can stay up to date on all these technologies by following him on LinkedIn and Twitter.