Zum Inhalt springen
Kitbay

ingesting-into-data-lake

Apache-2.0Aktualisiert am 11. Nov. 20263.464GitHub-Sterne

A skill in AWS's aws-data-analytics plugin for loading data into an AWS data lake.

Was der Skill vermittelt

A skill in AWS's aws-data-analytics plugin for loading data into an AWS data lake. Sources include S3 files or local uploads, JDBC databases such as Oracle, SQL Server, PostgreSQL, MySQL, RDS and Aurora, Amazon Redshift, Snowflake, BigQuery, DynamoDB and existing Glue tables being migrated. The default target is S3 Tables, with standard Iceberg on a general purpose bucket where S3 Tables is not used. It handles one-off loads, recurring pipelines and migrations: it checks the connection exists, clarifies the target, runs the source-specific workflow, validates the result and schedules repeat runs.

Funktionen

  • Load files and database tables into S3 Tables
  • Ingest data from Snowflake, BigQuery, Redshift or DynamoDB
  • Set up recurring ingestion pipelines
  • Convert existing tables to Iceberg