Configuring SFTP connectivity for ETL on Ambience/Repertoire Servers

Ambience 2020+ ETL Cookbook
, ,
1

This walkthrough describes how SFTP connectivity for ETL steps can be configured on both Ambience/Repertoire 202x Servers.

  1. Add the following configuration to etc/application.conf, edit accordingly to your environment. The fingerprint of the host machine is required to connect vis ssh. Multiple hosts and users can be configured:
ambience.etl.sftp {
  # Add multiple hosts following the structure below. The algorithm, format and fingerprint will be logged when
  # you try to connect and the fingerprint fails - add those into the config and restart to use the correct values
  hosts {
    #server {
    #  host: "changeme"
    #  port: 22
    #  algorithm: "EdDSA"
    #  format: "X.509"
    #  fingerprint: "MCowBQYDK2VwAyEAV7vHfoQ29yJbTj1rLivZ8Fh7+40utwmb2Ojx77/YX+4="
    #}
  }
  # Add multiple users, following the structure below. The group name ("default" here) is what is shown on the UI
  # The name field within the group (changeme here) is what is sent to the server when that group is chosen.
  # This allows multiple servers to be accessed with the same user name, but with different keys.
  users {
    #default {
    #  name: "changeme"
    #  keys: ${user.home}"/.ssh/id_rsa"
    #}
  }
}
  1. You will then be able to use the following SFTP steps in the ETL module:

  • SFTP List: lists the folder contents from an SFTP Server

  • SFTP Mkdirs: creates folders on an SFTP Server. Does not overwrite existing folders and supports creating multiple folders

  • SFTP Writer: collects the bytes from all incoming records and concatenates them into a single file which is written to the SFTP Server

  • SFTP Writer (Field): takes the bytes from an incoming record along with a file path and sends the bytes to an SFTP Server to be stored at the file path.

    Sample using the above ETL steps can be downloaded here (1.2 KB).