🔐 Secure Serverless Data Pipeline

A serverless data-processing pipeline built on AWS Lambda, S3, Step Functions, and API Gateway. It ingests files via an API endpoint, then orchestrates an Ingest → Clean → Transform workflow using AWS Step Functions.

---

📐 Architecture

┌──────────┐     ┌───────────┐     ┌──────────────────────┐
│  Client   │────▶│   API     │────▶│  API Upload Lambda   │
│  (curl /  │     │  Gateway  │     │  (writes to S3 raw/) │
│  Postman) │     └───────────┘     └──────────┬───────────┘
└──────────┘                                   │
                                               ▼
                              ┌────────────────────────────┐
                              │   S3 Bucket (encrypted)    │
                              │  raw/ → cleaned/ → final/  │
                              └────────────┬───────────────┘
                                           │ S3 Event
                                           ▼
                              ┌────────────────────────────┐
                              │   Trigger Lambda           │
                              │   (starts Step Function)   │
                              └────────────┬───────────────┘
                                           │
                              ┌────────────▼───────────────┐
                              │   Step Functions Pipeline  │
                              │  ┌────────┐                │
                              │  │ Ingest │                │
                              │  └───┬────┘                │
                              │      ▼                     │
                              │  ┌────────┐                │
                              │  │ Clean  │                │
                              │  └───┬────┘                │
                              │      ▼                     │
                              │  ┌───────────┐             │
                              │  │ Transform │             │
                              │  └───────────┘             │
                              └────────────────────────────┘

---

📁 Project Structure

secure-serverless-pipeline/
├── lambda/
│   ├── api_upload_lambda.py     # API Gateway → S3 upload handler
│   ├── trigger_function.py      # S3 event → Step Functions trigger
│   ├── ingest_lambda.py         # Step 1: Ingest raw data
│   ├── clean_lambda.py          # Step 2: Clean / sanitize data
│   ├── transform_lambda.py      # Step 3: Transform data
│   ├── deploy_lambda.sh         # Deployment script for Lambda functions
│   └── trigger_rules.json       # IAM policy for Step Functions access
├── workflow/
│   └── state_machine.json       # AWS Step Functions state machine definition
├── lambda_function.py           # Standalone S3 file processor (demo)
├── trust-policy.json            # IAM trust policy for Lambda execution role
├── encryption.json              # S3 server-side encryption config (AES-256)
├── requirements.txt             # Python dependencies
├── deploy.sh                    # Top-level deployment script
├── LICENSE                      # MIT License
└── README.md

---

🚀 Prerequisites

• AWS CLI configured with appropriate credentials
• Python 3.9+
• An S3 bucket with server-side encryption enabled (see encryption.json)
• IAM role with Lambda execution permissions (see trust-policy.json)

---

⚙️ Setup & Deployment

1. Install Python dependencies

pip install -r requirements.txt

2. Package Lambda functions

cd lambda
zip ingest.zip ingest_lambda.py
zip clean.zip clean_lambda.py
zip transform.zip transform_lambda.py
zip trigger_function.zip trigger_function.py
zip api_upload_lambda.zip api_upload_lambda.py

3. Deploy Lambda functions

chmod +x lambda/deploy_lambda.sh
./lambda/deploy_lambda.sh

4. Create the Step Functions state machine

aws stepfunctions create-state-machine \
  --name "SecureServerlessPipelineOrchestration" \
  --definition file://workflow/state_machine.json \
  --role-arn "arn:aws:iam::<ACCOUNT_ID>:role/lambda-execution-role" \
  --region eu-north-1

5. Configure S3 event trigger

Set up an S3 event notification on the raw/ prefix to invoke the trigger_function Lambda when new objects are created.

---

🔒 Security

• S3 encryption: AES-256 server-side encryption is enforced via encryption.json
• IAM least-privilege: Lambda functions use a dedicated execution role with a minimal trust policy
• No hardcoded secrets: Credentials are managed through AWS IAM and environment variables — never committed to source control

---

🧪 Testing

Invoke the API upload endpoint with a base64-encoded file:

curl -X POST https://<API_GATEWAY_URL>/upload \
  -H "Content-Type: application/json" \
  -d '{"file": "<BASE64_ENCODED_CONTENT>"}'

Or invoke a Lambda function directly:

aws lambda invoke \
  --function-name ingestLambda \
  --payload '{"file_key": "raw/test.txt"}' \
  --region eu-north-1 \
  output.json

---

📄 License

This project is licensed under the MIT License.