# acdc-aws-etl-pipeline
Infrastructure and code for the ACDC ETL pipeline and data operations in AWS

## Operating the pipeline

Day-to-day operations run through the `acdc` CLI. Install dependencies, then
explore the command surface and its built-in overview:

```bash
poetry install
poetry run acdc --help    # all command groups
poetry run acdc docs      # mental model, runbook, and prerequisites
```

New to the project? Start with the [CLI Operations Manual](docs/cli.md) — it is
the front door for running and inheriting this pipeline (mental model, runbooks,
EC2 long jobs, safety rails, and the AWS Glue jobs that live outside the CLI). For
an exhaustive listing of every command and flag with examples, see the
[CLI Command Reference](docs/cli-reference.md).

## Documentation

### Core Configuration & Management
- [Deployment Configuration Guide](docs/config.md)
- [Dictionary Deployment](docs/dictionary_deployment.md)
- [Service Management](docs/service_management.md)
- [Kubernetes Utilities](docs/k8s_utilities.md)
- [Troubleshooting](docs/troubleshooting.md)

### Data Lifecycle & ETL
- [Data Ingestion](docs/data_ingestion.md)
- [Data Validation](docs/data_validation.md)
- [Data Transformation (dbt)](docs/data_transformation_dbt.md)
- [Data Releases](docs/write_data_release.md)
- [Data Deletion](docs/data_deletion.md)
- [Metadata Deletion by GUID](docs/metadata_deletion_by_guid.md)

### Metadata & Registry Operations
- [REST API Upload to Sheepdog](docs/rest_api_sheepdog_upload.md)
- [IndexD File Registration](docs/indexd_registration.md)
- [Synthetic Data Generation](docs/synthetic_data_generation.md)

### Analysis & Querying
- [Querying Athena](docs/querying_athena.md)
- [Writing Athena Queries to JSON](docs/write_athena_queries_to_json.md)

## Library and source code (`src/acdc_aws_etl_pipeline`)

The Python package in [`src/acdc_aws_etl_pipeline`](src/acdc_aws_etl_pipeline) provides reusable utilities for ingestion, validation, uploads, and Athena/Glue operations used across the pipeline and services.

### Modules

- **`ingest/`**: ingestion helpers for loading source datasets into S3/Glue (see [`ingest/ingest.py`](src/acdc_aws_etl_pipeline/ingest/ingest.py)).
- **`upload/`**: Gen3/Sheepdog metadata submission and deletion utilities (e.g. [`upload/metadata_submitter.py`](src/acdc_aws_etl_pipeline/upload/metadata_submitter.py)).
- **`validate/`**: schema validation utilities and helpers for validation workflows (see [`validate/validate.py`](src/acdc_aws_etl_pipeline/validate/validate.py)).
- **`utils/`**: shared Athena/Glue/dbt/release helpers (e.g. [`utils/athena_utils.py`](src/acdc_aws_etl_pipeline/utils/athena_utils.py), [`utils/release_writer.py`](src/acdc_aws_etl_pipeline/utils/release_writer.py)).

### Local development

To install dependencies and run tests:

```bash
pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
poetry run pytest
```

### Install from PyPI

Releases are published automatically, so you can also install the package directly:

```bash
pip install acdc_aws_etl_pipeline