Skip to content
Kitbay

ingesting-into-data-lake

Apache-2.0Updated on Nov 11, 20263,464GitHub stars

A skill in AWS's aws-data-analytics plugin for loading data into an AWS data lake.

What the skill teaches

A skill in AWS's aws-data-analytics plugin for loading data into an AWS data lake. Sources include S3 files or local uploads, JDBC databases such as Oracle, SQL Server, PostgreSQL, MySQL, RDS and Aurora, Amazon Redshift, Snowflake, BigQuery, DynamoDB and existing Glue tables being migrated. The default target is S3 Tables, with standard Iceberg on a general purpose bucket where S3 Tables is not used. It handles one-off loads, recurring pipelines and migrations: it checks the connection exists, clarifies the target, runs the source-specific workflow, validates the result and schedules repeat runs.

Capabilities

  • Load files and database tables into S3 Tables
  • Ingest data from Snowflake, BigQuery, Redshift or DynamoDB
  • Set up recurring ingestion pipelines
  • Convert existing tables to Iceberg