Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,24 @@ git push # LFS objects upload to your bucket via git-los

## Documentation

### User Guides

- [Installation](./docs/installation.md) — Installing and setting up git-los
- [Configuration](./docs/configuration.md) — Backend setup and options
- [Troubleshooting](./docs/troubleshooting.md) — Common issues and solutions

### Example Workflows

- [Game Development](./docs/examples/game-dev.md) — Managing textures, models, and audio
- [ML Datasets](./docs/examples/ml-datasets.md) — Dataset versioning and model management

### Development

- [Contributing](./CONTRIBUTING.md) — How to contribute to git-los
- [Development Guide](./docs/development.md) — Building and testing locally

### Specifications

See the [spec](./spec/README.md) directory for detailed specifications:

- [Architecture](./spec/architecture.md) — System components and their interactions
Expand Down
278 changes: 278 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,278 @@
# Configuration Guide

This guide covers all configuration options for git-los, including cloud backend setup, daemon settings, and advanced options.

## Quick Start

### Google Cloud Storage (GCS)

```bash
# Set up Application Default Credentials
gcloud auth application-default login

# Configure a repository
cd your-repo
git-los remote add origin gs://your-bucket/lfs
```

### Amazon S3

```bash
# Set up AWS credentials
aws configure

# Configure a repository
cd your-repo
git-los remote add origin s3://your-bucket/lfs
```

## Backend URL Format

git-los uses URL-style identifiers for storage backends:

| Backend | URL Format | Example |
|---------|------------|---------|
| GCS | `gs://bucket/prefix` | `gs://my-project-lfs/objects` |
| S3 | `s3://bucket/prefix` | `s3://my-lfs-bucket/data` |

The prefix is optional but recommended for organizing LFS objects.

## Cloud Credentials

### Google Cloud Storage

git-los uses Application Default Credentials (ADC) in this order:

1. `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to a service account key
2. User credentials from `gcloud auth application-default login`
3. Service account attached to GCE/GKE instances
4. Workload Identity on GKE

**For development:**
```bash
gcloud auth application-default login
```

**For CI/CD:**
```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
```

**Required IAM permissions:**
- `storage.objects.create` - Upload objects
- `storage.objects.get` - Download objects
- `storage.objects.delete` - Delete objects (optional)

### Amazon S3

git-los uses standard AWS credential chain:

1. Environment variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
2. AWS credentials file (`~/.aws/credentials`)
3. IAM role attached to EC2/ECS instances
4. Web Identity Token for EKS

**For development:**
```bash
aws configure
```

**For CI/CD:**
```bash
export AWS_ACCESS_KEY_ID=your-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-key
export AWS_REGION=us-east-1
```

**Required IAM permissions:**
```json
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::your-bucket/lfs/*"
}
```

## Git Configuration

### Repository-Level Settings

Configure in your repository's `.git/config`:

```gitconfig
[lfs]
url = gs://your-bucket/lfs
```

Or use the CLI:
```bash
git-los remote add origin gs://your-bucket/lfs
```

### Global Settings

Configure in `~/.gitconfig`:

```gitconfig
[lfs]
standalonetransferagent = git-los

[lfs "customtransfer.git-los"]
path = git-los
args = agent
concurrent = true
direction = both
```

These are automatically set by `git-los install --global`.

## Daemon Configuration

The daemon provides connection pooling and credential caching.

### Starting the Daemon

```bash
# Start in foreground (for debugging)
git-los daemon start --foreground

# Start in background (normal operation)
git-los daemon start

# Check status
git-los daemon status
```

### Daemon Flags

| Flag | Description | Default |
|------|-------------|---------|
| `--foreground` | Run in foreground | `false` |
| `--log-level` | Log level (debug, info, warn, error) | `info` |
| `--log-file` | Log file path | `stderr` |
| `--max-concurrent` | Max concurrent transfers | `8` |

### Socket Location

The daemon socket is created at:
1. `$GIT_LOS_SOCKET` if set
2. `$XDG_RUNTIME_DIR/git-los/daemon.sock`
3. `~/.cache/git-los/daemon.sock`

### Log Files

Daemon logs are written to:
- Default: `stderr` (when running in foreground)
- Configured: Use `--log-file /path/to/daemon.log`

Logs are rotated automatically:
- Max size: 10MB per file
- Max backups: 3 files
- Compression: gzip

## Environment Variables

| Variable | Description |
|----------|-------------|
| `GIT_LOS_SOCKET` | Override daemon socket path |
| `GIT_LOS_DEBUG` | Enable debug logging (`1` or `true`) |
| `GOOGLE_APPLICATION_CREDENTIALS` | GCS service account key path |
| `AWS_ACCESS_KEY_ID` | AWS access key |
| `AWS_SECRET_ACCESS_KEY` | AWS secret key |
| `AWS_REGION` | AWS region |

## Advanced Options

### Custom S3 Endpoints

For S3-compatible storage (MinIO, etc.):

```bash
export AWS_ENDPOINT_URL=http://localhost:9000
git-los remote add origin s3://bucket/prefix
```

### Concurrent Transfers

Adjust the number of concurrent transfers:

```gitconfig
[lfs "customtransfer.git-los"]
concurrent = true
```

### Retry Configuration

git-los automatically retries failed transfers with exponential backoff:
- Max attempts: 3
- Initial wait: 1 second
- Max wait: 30 seconds

Retryable errors include:
- Network timeouts
- Server errors (5xx)
- Rate limiting

Non-retryable errors include:
- Authentication failures (401, 403)
- Not found (404)
- Bad requests (400)

## Configuration Precedence

git-los reads configuration in this order (later overrides earlier):

1. Built-in defaults
2. System git config (`/etc/gitconfig`)
3. Global git config (`~/.gitconfig`)
4. Repository git config (`.git/config`)
5. Environment variables
6. Command-line flags

## Troubleshooting Configuration

### Verify Backend URL

```bash
git config lfs.url
# or
git-los remote list
```

### Check Credentials

**GCS:**
```bash
gcloud auth application-default print-access-token
```

**S3:**
```bash
aws sts get-caller-identity
```

### Test Connection

```bash
# Upload a test file
echo "test" > /tmp/test.txt
git-los upload --oid $(sha256sum /tmp/test.txt | cut -d' ' -f1) --path /tmp/test.txt
```

### View Daemon Logs

```bash
git-los daemon status
# Check log file location, then:
tail -f /path/to/daemon.log
```

## Next Steps

- [Troubleshoot common issues](./troubleshooting.md)
- [Game development workflow](./examples/game-dev.md)
- [ML dataset management](./examples/ml-datasets.md)
Loading
Loading