Architecture: Topic Relationships, Word Forms & Disambiguation

# Architecture: Topic Relationships, Word Forms & Disambiguation

**Depends on:** #48 (MVP: Devil's Dictionary Phoenix Rebuild)

## The Problem

How do we handle relationships between similar words and concepts in a way that's simple for MVP but won't create technical debt?

**Key Scenarios:**
1. **Word Forms**: aristocracy → aristocratic → aristocrat (same root, different grammar)
2. **Synonyms**: truth, verity, veracity (different words, same meaning)
3. **Related Concepts**: truth → honesty, lie, reality (semantic connections)
4. **Disambiguation**: "bank" (financial) vs "bank" (river edge)
5. **Empty Topic Fallback**: if "veracity" has no definitions, what do we show?

---

## Analysis: Relationship Types

### Type 1: Morphological Variants (Word Forms)
Same root word, different grammatical forms:
- aristocracy (noun) → aristocratic (adj) → aristocrat (noun/person)
- truth → truthful → truthfully

**Characteristic**: May each have their own definitions (Bierce defined many word forms separately), but should cross-reference.

### Type 2: Synonyms
Different words, essentially same meaning:
- truth, verity, veracity
- lie, falsehood, untruth

**Characteristic**: One may be "canonical" (more common), others might redirect or show "see also."

### Type 3: Semantic Relations
Different concepts that are connected:
- **Antonyms**: truth ↔ lie
- **Broader/Narrower**: democracy, aristocracy, monarchy → (all are) government types
- **Associated**: truth → honesty → integrity

**Characteristic**: Each is a distinct concept; relationships are informational, not redirects.

### Type 4: Disambiguation
Same word with completely different meanings:
- bank (financial institution) vs bank (river edge)
- crane (bird) vs crane (machine)

**Characteristic**: Must be separate topics with a disambiguation landing page.

### Type 5: Pure Redirects (Aliases)
Alternative names for the same thing:
- USA → United States
- NYC → New York City

**Characteristic**: Hard redirect, user never sees the alias page.

---

## Recommended Schema

### Modified Topics Table

```elixir
schema "topics" do
  field :title, :string
  field :slug, :string
  field :type, Ecto.Enum, values: [:concept, :thing, :place, :event, :action, :person, :organization, :other]
  
  # NEW: Display handling
  field :display_type, Ecto.Enum, 
    values: [:standard, :redirect, :disambiguation], 
    default: :standard
  
  # NEW: For pure redirects only (hard redirect, URL changes)
  belongs_to :redirect_to, Topic
  
  # Existing fields...
  field :pronunciation, :string
  field :part_of_speech, :string
  field :definition_count, :integer, default: 0
  
  timestamps()
end
```

### Topic Relationships Table

```elixir
schema "topic_relationships" do
  belongs_to :topic, Topic
  belongs_to :related_topic, Topic
  
  field :relationship_type, Ecto.Enum, values: [
    # MVP (Phase 1)
    :word_form,      # aristocracy ↔ aristocratic (morphological variants)
    :synonym,        # truth ↔ verity (same meaning, different word)
    :related,        # truth → honesty (semantically connected)
    :see_also,       # loose connection, editorial choice
    
    # Phase 2+
    :antonym,        # truth ↔ lie (opposites)
    :broader,        # aristocracy → government (hypernym)
    :narrower,       # government → aristocracy (hyponym)
    :disambiguates   # bank → bank_finance, bank_geography
  ]
  
  # Is this relationship symmetric? (synonym = yes, broader = no)
  field :bidirectional, :boolean, default: true
  
  # For ordering related topics by relevance
  field :weight, :float, default: 1.0
  
  timestamps()
end
```

---

## UI Patterns by Relationship Type

### 1. Word Form (aristocracy ↔ aristocratic)

**Topic with definitions:**
```
┌────────────────────────────────────────────────────┐
│  ARISTOCRATIC, adj.                                │
│  [definitions...]                                  │
│                                                    │
│  ─────────────────────────────────────────────────│
│  Word Family: aristocracy • aristocrat             │
└────────────────────────────────────────────────────┘
```

**Topic WITHOUT definitions (empty word form):**
```
┌────────────────────────────────────────────────────┐
│  ARISTOCRATICAL, adj.                              │
│  /əˌrɪstəˈkrætɪkəl/                               │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  An archaic form of ARISTOCRATIC.            │ │
│  │                                              │ │
│  │  → See ARISTOCRATIC for definitions          │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  Word Family: aristocracy • aristocratic           │
└────────────────────────────────────────────────────┘
```

### 2. Synonym (truth ↔ verity)

**Topic with definitions:**
```
┌────────────────────────────────────────────────────┐
│  TRUTH, n.                                         │
│  [definitions...]                                  │
│                                                    │
│  ─────────────────────────────────────────────────│
│  Synonyms: verity • veracity • sooth               │
└────────────────────────────────────────────────────┘
```

**Topic WITHOUT definitions (empty synonym):**
```
┌────────────────────────────────────────────────────┐
│  VERITY, n.                                        │
│  /ˈverɪti/                                        │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  A formal synonym of TRUTH.                  │ │
│  │                                              │ │
│  │  → See TRUTH for definitions                 │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  Synonyms: truth • veracity                        │
└────────────────────────────────────────────────────┘
```

### 3. Related Topics

**Sidebar section (all topics):**
```
┌────────────────────────────────────────────────────┐
│  Related Topics                                    │
│  ─────────────                                     │
│  • honesty                                         │
│  • reality                                         │
│  • fact                                            │
│  • sincerity                                       │
└────────────────────────────────────────────────────┘
```

### 4. Disambiguation Page

**When topic has display_type = :disambiguation:**
```
┌────────────────────────────────────────────────────┐
│  BANK                                              │
│                                                    │
│  Bank may refer to:                                │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  BANK (finance)                              │ │
│  │  A financial institution that accepts        │ │
│  │  deposits and makes loans.                   │ │
│  │  → 5 definitions                             │ │
│  └──────────────────────────────────────────────┘ │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  BANK (geography)                            │ │
│  │  The land alongside a river or body of       │ │
│  │  water.                                      │ │
│  │  → 2 definitions                             │ │
│  └──────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────┘
```

### 5. Pure Redirect

**User visits /topics/usa → automatically redirects to /topics/united-states**

No page rendered for redirect topics. HTTP 301 redirect.

---

## Empty Topic Fallback Logic

```elixir
def render_topic(topic) do
  cond do
    # Hard redirect
    topic.display_type == :redirect && topic.redirect_to ->
      redirect_to(topic.redirect_to)
    
    # Disambiguation page
    topic.display_type == :disambiguation ->
      render_disambiguation(topic)
    
    # Has own definitions
    topic.definition_count > 0 ->
      render_standard_topic(topic)
    
    # No definitions, but has word_form or synonym with definitions
    related = find_related_with_definitions(topic, [:word_form, :synonym]) ->
      render_see_also_topic(topic, related)
    
    # Truly empty
    true ->
      render_empty_topic(topic)
  end
end

defp find_related_with_definitions(topic, relationship_types) do
  topic
  |> get_relationships(relationship_types)
  |> Enum.filter(& &1.related_topic.definition_count > 0)
  |> Enum.sort_by(& &1.weight, :desc)
  |> List.first()
end
```

---

## Populating Relationships

### Phase 1: Manual Curation (MVP)
- Admin creates relationships when adding topics
- Parse cross-references from Bierce ("See X" patterns)

### Phase 2: API Integration
We already planned ConceptNet and Datamuse integrations:

| Source | Provides |
|--------|----------|
| **ConceptNet** | /r/RelatedTo, /r/Synonym, /r/Antonym, /r/FormOf |
| **Datamuse** | rel_syn (synonyms), rel_trg (triggers/associations) |

**Auto-population workflow:**
1. When topic is created/enriched, query ConceptNet
2. Find existing topics that match related concepts
3. Create relationship records with appropriate types
4. Human review/approval for quality control

### Phase 3: User Contributions
- Users can suggest relationships
- Moderation queue before publishing

---

## Validation Rules

### Prevent Circular Redirects
```elixir
def changeset(topic, attrs) do
  topic
  |> cast(attrs, [:redirect_to_id])
  |> validate_no_circular_redirect()
end

defp validate_no_circular_redirect(changeset) do
  # redirect_to cannot point to a topic that itself redirects
  # prevents: A → B → C → A
end
```

### Disambiguation Requirements
```elixir
def validate_disambiguation(topic) do
  if topic.display_type == :disambiguation do
    disambiguates_count = count_relationships(topic, :disambiguates)
    if disambiguates_count < 2 do
      {:error, "Disambiguation topics must have at least 2 disambiguates relations"}
    end
  end
end
```

### Bidirectional Consistency
```elixir
def create_relationship(topic_a, topic_b, type, opts \\ []) do
  bidirectional = Keyword.get(opts, :bidirectional, default_bidirectional(type))
  
  Multi.new()
  |> Multi.insert(:forward, %TopicRelationship{
    topic_id: topic_a.id,
    related_topic_id: topic_b.id,
    relationship_type: type,
    bidirectional: bidirectional
  })
  |> Multi.insert(:reverse, fn _ ->
    if bidirectional do
      %TopicRelationship{
        topic_id: topic_b.id,
        related_topic_id: topic_a.id,
        relationship_type: inverse_type(type),
        bidirectional: bidirectional
      }
    end
  end)
  |> Repo.transaction()
end

defp inverse_type(:broader), do: :narrower
defp inverse_type(:narrower), do: :broader
defp inverse_type(type), do: type  # symmetric types
```

### Relationship Limits
```elixir
@max_synonyms 10
@max_word_forms 5
@max_related 20

def validate_relationship_limits(topic, new_type) do
  current_count = count_relationships(topic, new_type)
  max = max_for_type(new_type)
  
  if current_count >= max do
    {:error, "Maximum #{max} #{new_type} relationships allowed"}
  end
end
```

---

## MVP Scope

### ✅ Phase 1 (MVP)

**Schema:**
- [ ] Add `display_type` to topics (default :standard)
- [ ] Add `redirect_to_id` to topics
- [ ] Create `topic_relationships` table with MVP types

**Relationship Types:**
- [ ] `word_form` - morphological variants
- [ ] `synonym` - same meaning
- [ ] `related` - semantic connection
- [ ] `see_also` - editorial connection

**UI:**
- [ ] Word Family section on topic pages
- [ ] Synonyms section on topic pages
- [ ] Related Topics sidebar
- [ ] Empty topic fallback to related word/synonym

**Validation:**
- [ ] No circular redirects
- [ ] Bidirectional relationship creation

### ❌ Phase 2+

**Schema:**
- [ ] Add disambiguation display type handling

**Relationship Types:**
- [ ] `antonym` - opposites
- [ ] `broader` / `narrower` - hypernym/hyponym
- [ ] `disambiguates` - disambiguation

**Features:**
- [ ] Disambiguation pages
- [ ] ConceptNet auto-population
- [ ] Datamuse auto-population
- [ ] Antonyms UI section
- [ ] Broader/Narrower taxonomy navigation

---

## Why This Won't Haunt Us

1. **Single relationships table** - Not multiple join tables for different relationship types
2. **Enum for types** - Easy to extend without schema changes
3. **Bidirectional flag** - Handles symmetric (synonym) and asymmetric (broader) relations
4. **Weight field** - Future-proofs ordering and relevance
5. **Clear UI patterns** - Each relationship type has defined behavior
6. **Explicit empty handling** - No magic, user sees "See X" messages
7. **Validation rules** - Prevents circular redirects and orphaned disambiguation
8. **Migration path** - MVP types are subset of full types

---

## Open Questions

1. **Should word forms share definitions by default?**
   - Recommendation: No, each topic owns its definitions. UI shows cross-references.

2. **How do we handle Bierce's cross-references during seed?**
   - Recommendation: Parse "See X" patterns, create `see_also` relationships

3. **Should redirects be soft (show page) or hard (HTTP 301)?**
   - Recommendation: Hard redirects for aliases, soft "see X" for synonyms/word forms

4. **How do we handle conflicting auto-generated relationships?**
   - Recommendation: Queue for human review, don't auto-publish

---

## References

- #48 - MVP: Devil's Dictionary Phoenix Rebuild
- [WordNet](https://wordnet.princeton.edu/) - Linguistic database model
- [ConceptNet](https://conceptnet.io/) - Planned API integration
- [Wikipedia Disambiguation](https://en.wikipedia.org/wiki/Wikipedia:Disambiguation) - UI patterns

---

*"The relationships between words are as important as the words themselves."*

Source	Provides
ConceptNet	/r/RelatedTo, /r/Synonym, /r/Antonym, /r/FormOf
Datamuse	rel_syn (synonyms), rel_trg (triggers/associations)

Uh oh!

Architecture: Topic Relationships, Word Forms & Disambiguation #60

Description

Architecture: Topic Relationships, Word Forms & Disambiguation

The Problem

Analysis: Relationship Types

Type 1: Morphological Variants (Word Forms)

Type 2: Synonyms

Type 3: Semantic Relations

Type 4: Disambiguation

Type 5: Pure Redirects (Aliases)

Recommended Schema

Modified Topics Table

Topic Relationships Table

UI Patterns by Relationship Type

1. Word Form (aristocracy ↔ aristocratic)

2. Synonym (truth ↔ verity)

3. Related Topics

4. Disambiguation Page

5. Pure Redirect

Empty Topic Fallback Logic

Populating Relationships

Phase 1: Manual Curation (MVP)

Phase 2: API Integration

Phase 3: User Contributions

Validation Rules

Prevent Circular Redirects

Disambiguation Requirements

Bidirectional Consistency

Relationship Limits

MVP Scope

✅ Phase 1 (MVP)

❌ Phase 2+

Why This Won't Haunt Us

Open Questions

References

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions