This recipe is an example of how to design a simple ETL chainset for the step, “Array Paginate”.
Background Information:
- This step converts each record in holding an array into a series of records out, each holding a slice from the array, where the length of the slice is limited to a fixed page size.
- More information can be found in the ETL dictionary.
Prerequisite:
- A dataset which consists of an Array field.
Notes:
- This example uses a JSON data structure
ETL Array Paginate
-
Create a Sample JSON data which includes an Array Field locally & save it as a .json file
For example, sample Data Used: [sampledata2.json]
{"color":"red", "shades":["cherry", "rose", "mahogany", "crimson", "garnet", "wine", "currant", "ruby", "burgundy"]} -
Create a new ETL Chainset.
For example,
-
Add a new Blank Chain.
For example,
-
Add the Step, “URL Reader” & input in your .json file path over in the “URL” field.
For example,
Expand the icon on the top right corner for more details on this step.
Notes:
- Do ensure that the path given in the “URL” field is declared as readable in your Ambience Server “application.conf” file.
- This setting can be found under “elixir.data.url”
-
Add the Step, “JSON Reader”
-
Add the Step, “Array Paginate” & configure the necessary parameters.
For example,
Expand the icon on the top right corner for more details on this step. -
Run the ETL Steps.
Output:
// Initial Data
// After running the ETL chain
[As configured in the “Page Size” for Array Paginate ETL step, each record only holds a maximum of 2 Array Slices.]