Adding and Reading Talend Headers and Trailers
Adding a header and trailer to a file
Let us look at how to add a custom header, footer to file. This is often required for file processing and validation.
Requirement: Create a delimited file with below structure.
File Name: Name of the actual File.
PID : process ID.
File Created Date: Date & Time.
Number of Records: record count.
This is our final job design.
Reading headers and trailers using tMap
This recipe shows how to parse a file that has a header and trailer records, and a record type at the start of a line.
Open the jo_cook_ch08_0060_headTrailtMap job.
How to accomplish it…
The steps for reading headers and trailers using tMap are as follows:
- Drag a tMap component onto the canvas.
- Connect the tFileInputFullRow to tMap, and rename the flow to customerIn.
- Open tMap, and create three new outputs. Name them header, detail, and trailer.
- Copy the input field line into each of the new outputs.
- Add the expression filter line.startsWith(“00”) to the header output table.
- Add the expression filter line.startsWith(“01”) to the detail output table.
- Add the expression filter line.startsWith(“99”) to the trailer output table.
- Your tMap should now look like the one shown as follows:
- Close tMap, and drag three tExtractDelimitedFields components to the canvas, along with three tLogRow
- Join each output from tMap to each of the tExtractDelimitedFields components.
- Change the delimiter in each of the tExtractDelimitedFields components to comma (,).
- Open the tLogRow components, and assign each one a schema from those listed, as follows. This can be easily done by dragging the metadata onto the tLogRow component as described in, Metadata and Schemas.
sc_cook_ch8_0060_genericCustomerHeader sc_cook_ch8_0060_genericCustomerDetail sc_cook_ch8_0060_genericCustomerTrailer
- Link the tExtractDelimitedFields to the tLogRows, making sure that you accept the output schema.
- Your job should now look like this:
- Change the tLogRow components to the output Mode of Vertical, and run the job.
How it works…
tFileInputFullRow allows us to read a row of any format into tMap. This is important, because we do not want records to be rejected due to schema errors at this stage.
The start of each row is then tested for the record type; 00, 01, or 02, the header, detail, or trailer records respectively.
The different rows are then passed to a tExtractDelimitedFields component for breaking down into the individual schema columns.
This isn’t the only method of reading files with headers and trailers, and in fact, the best Talend method would be to use the tFileInputMSDelimited component, for this example.
This method, however, is much more flexible, in which the conditions for sending in the data as an output to each of the flows does not depend upon a fixed field being present.
Reading headers and trailers with no identifiers
This recipe shows how to parse a file that has a header and trailer records, but does not have an associated record type. Instead, the header is the first record in the file, and the trailer is the last record in the file.
Open the jo_cook_ch08_0070_headTrailtMapNoType job. You will see that it is a slightly changed version of the completed job from the previous recipe; the output schemas have changed.
How to achieve it…
The steps for reading headers and trailers with no identifiers are as follows:
- Drag a tFileRowCount component onto the canvas.
- Open the tFileRowCount, and change File Name to
which is the same as our input file.
- Connect an onSubJobOk trigger from the tFileRowCount component to the tFileInputDelimited.
- Open the tMap, and add a new variable rowCount. Set its expression to Numeric.sequence(“rowNumber”,1,1).
- Change the Filter expressions for header, detail, and trailer to those shown as follows:
Var.rowNumber == 1
- Set the detail and trailer output options to Catch output rejects.
- Your tMap should now look like this:
- Run the job, and you should see the individual row types being printed.
How it works…
The tFileRowCount component tells us how many rows are present in the file.
In the tMap, we use a sequence to calculate the current line number. If the line number is 1, then we have a header row. If it is equal to the row count (held in globalMap), then we have a trailer row, and all other rows are detail rows.
We then use the tExtractDelimitedFields to extract the individual delimited fields into a different schema for each of the row types.