Lexiconic

Lexiconic is a digital exploration of untranslatable words - terms from various languages that express concepts, feelings, or situations that lack a direct translation in English. Each word includes:

• Multiple viewing modes: List, grid, and interactive map visualization
• Semantic search: Search by meaning, not just keywords
• Audio pronunciations: Hear how each word is pronounced
• Flexible sorting: Alphabetical, reverse, or randomized (with consistent seeds)
• Responsive design: Beautiful interface across all devices
• Rich word data: Includes categories, literal translations, and usage notes

Words are stored in two separate files for optimal performance:

/public/data/words.json - All word metadata (fast initial load ~236KB):

• Word in native script and romanization
• Language, language family, and geographic location
• Definition, literal meaning, and usage notes
• Pronunciation audio file references (stored in /public/pronunciations/)

/public/data/embeddings.json - Pre-computed embeddings only (~14MB):

• Map of word → embedding vector + hash
• Loaded in background via SWR for semantic search
• Separate to keep words.json small and editable

The app uses AI embeddings to enable semantic search:

1. Each word's content is converted to a vector embedding (stored separately in embeddings.json)
2. When you search, your query is also embedded
3. Results are ranked by cosine similarity to find conceptually related words
4. Embeddings are pre-computed and loaded in background for fast searches
5. Word metadata loads instantly (~236KB), embeddings load progressively (~14MB)

• Framework: Next.js 15 with React 19
• Styling: Tailwind CSS with shadcn/ui components
• Maps: Mapbox GL for geographic visualization
• Search: Client-side semantic search with SWR for data fetching
• Audio: OpenAI TTS for pronunciation generation

• Node.js 18+ and npm
• (Optional) OpenAI API key for pronunciation generation

This project is currently configured to be hosted at /lexiconic (not at the root).

If you want to run it locally at the root path (recommended for local development), you'll need to make the following changes:

1. Remove the basePath in next.config.mjs:

   // Change this:
   basePath: '/lexiconic',
   
   // To this:
   // basePath: '', // or remove the line entirely

2. Update hardcoded paths in the codebase:

   • app/api/og/route.tsx - Remove /lexiconic from font and image paths
   • Any other references to /lexiconic in your code

3. Update environment variables:

   # For local development, you can omit NEXT_PUBLIC_VERCEL_URL
   # or set it to your local URL:
   NEXT_PUBLIC_VERCEL_URL=http://localhost:3000

1. Clone the repository

   git clone <repository-url>
   cd lexiconic

2. Install dependencies

   npm install

3. Configure for local development (see "Important: /lexiconic Base Path" above)

   • Comment out or remove basePath: '/lexiconic' in next.config.mjs
   • Search for /lexiconic in the codebase and update paths as needed

4. Set up environment variables Create a .env.local file:

   # Required for pronunciation generation
   OPENAI_API_KEY=your_openai_api_key_here
   
   # Required for map view functionality
   NEXT_PUBLIC_MAPBOX_TOKEN=your_mapbox_token_here
   
   # Required for running certain scripts (generate-definition, generate-phonetic)
   BRAINTRUST_API_KEY=your_braintrust_api_key_here
   
   # Optional - defaults to localhost:3000 in development
   NEXT_PUBLIC_VERCEL_URL=http://localhost:3000

   Getting API Keys:

   • OpenAI: Get from platform.openai.com
   • Mapbox: Get from mapbox.com/account/access-tokens
   • Braintrust: Get from braintrust.dev

   Note: The map view won't work without NEXT_PUBLIC_MAPBOX_TOKEN. Pronunciation generation and some scripts won't work without their respective API keys.

5. Run the development server

   npm run dev

6. Open your browser

   • If you removed the basePath: http://localhost:3000
   • If you kept the basePath: http://localhost:3000/lexiconic

• npm run dev - Start development server
• npm run build - Build for production
• npm start - Run production build
• npm run lint - Run ESLint
• npm run add-word "word" "Language" "source" - Add a new word with automatic generation
• npm run generate - Generate all missing metadata, pronunciations, and embeddings
• npm run generate-metadata - Generate metadata for words missing it
• npm run generate-pronunciation - Generate audio for words missing it
• npm run generate-embedding - Generate embeddings for words with changed content

The easiest way to add a word is to use the CLI tool:

npm run add-word "saudade" "Portuguese" "https://example.com/source"

What it does:

1. Validates inputs and checks for duplicates
2. Creates a temporary backup of words.json
3. Generates comprehensive metadata via Braintrust AI
4. Generates pronunciation audio via OpenAI TTS
5. Generates semantic embeddings via OpenAI
6. Rolls back on error or cleans up backup on success

Then commit:

git add .
git commit -m "add saudade"
git push

If you prefer to manually edit the JSON:

1. Add minimal entry to /public/data/words.json:

{
  "word": "saudade",
  "language": "Portuguese",
  "source": "https://example.com/source"
}

2. Run generation (processes only what's needed):

npm run generate

This runs three scripts in sequence, each processing only words that need processing:

• generate-metadata - Fills in missing metadata fields
• generate-pronunciation - Creates missing audio files
• generate-embedding - Generates missing/changed embeddings

3. Commit the changes:

git add .
git commit -m "add saudade"

If you have your own metadata but want to generate only pronunciation and embeddings:

1. Add complete metadata entry to /public/data/words.json:

{
  "word": "saudade",
  "language": "Portuguese",
  "source": "https://example.com/source",
  "family": "Indo-European",
  "category": "Emotion",
  "definition": "A deep emotional state of nostalgic or profound melancholic longing for something or someone that one cares for and/or loves.",
  "literal": "Longing, missing, nostalgia",
  "usage_notes": "Often associated with Portuguese and Brazilian culture, expressing the feeling of missing something that may never return.",
  "english_approx": "nostalgia, longing, yearning",
  "phonetic": "saw-ˈdɑ-dʒi",
  "location": "Portugal",
  "lat": 39.3999,
  "lng": -8.2245
}

2. Generate only pronunciation and embeddings:

# Generate just pronunciation audio
npm run generate-pronunciation

# Generate just embeddings
npm run generate-embedding

3. Commit the changes:

git add .
git commit -m "add saudade"

If you want to regenerate metadata, pronunciations, or embeddings for all words (e.g., after updating prompts):

# Regenerate all metadata
npm run generate-metadata -- --all

# Regenerate all pronunciations
npm run generate-pronunciation -- --all

# Regenerate all embeddings
npm run generate-embedding -- --all

The scripts/deprecated/ directory contains granular scripts for regenerating individual metadata fields. These are useful when you need to update or fix specific fields without regenerating all metadata:

Available Scripts:

• generate-definition.ts - Regenerates only the definition field
• generate-phonetic.ts - Regenerates only the phonetic field
• generate-usage-note.ts - Regenerates only the usage_notes field
• generate-location.ts - Regenerates only the location field

Requirements:

• BRAINTRUST_API_KEY must be set in .env.local
• All scripts create automatic backups in /public/data/backup/ before making changes (deleted on success)

Usage Examples:

# Regenerate definitions for words missing this field only (default)
npx tsx scripts/deprecated/generate-definition.ts

# Regenerate definitions for ALL words (force regeneration)
npx tsx scripts/deprecated/generate-definition.ts -- --all

# Regenerate phonetics for words missing this field only (default)
npx tsx scripts/deprecated/generate-phonetic.ts

# Regenerate phonetics for ALL words (force regeneration)
npx tsx scripts/deprecated/generate-phonetic.ts -- --all

# Regenerate usage notes for words missing this field only (default)
npx tsx scripts/deprecated/generate-usage-note.ts

# Regenerate usage notes for ALL words (force regeneration)
npx tsx scripts/deprecated/generate-usage-note.ts -- --all

# Regenerate locations for words missing this field only (default)
npx tsx scripts/deprecated/generate-location.ts

# Regenerate locations for ALL words (force regeneration)
npx tsx scripts/deprecated/generate-location.ts -- --all

When to use:

• After updating a specific Braintrust prompt
• To fix incorrect data in a specific field
• To add a field to words that are missing it
• When testing changes to field generation logic

Note: These scripts are considered "deprecated" because the main workflow uses generate-metadata.ts which handles all fields at once. However, they remain useful for targeted updates.

lexiconic/
├── app/                    # Next.js app directory
│   ├── page.tsx           # Main page with server-side sorting
│   ├── layout.tsx         # Root layout
│   └── api/               # API routes for search, pronunciations, etc.
├── braintrust/            # Braintrust prompts
├── components/            # React components
│   ├── words-client.tsx   # Main client component
│   ├── words-list.tsx     # List view
│   ├── map-view.tsx       # Map visualization
│   └── ...                # Other UI components
├── hooks/                 # Custom React hooks
│   ├── use-device-type.ts # Device type detection
│   └── use-url-state.ts   # URL state management
├── lib/                   # Utility functions
│   ├── load-words.ts      # Server-side word loading
│   └── semantic-search.ts # Embedding and search logic
├── public/
│   ├── data/
│   │   ├── words.json           # Word metadata (no embeddings)
│   │   └── embeddings.json      # Embeddings map (word → vector)
│   ├── images/
│   │   └── og.png               # Open Graph image
│   └── pronunciations/          # MP3 pronunciation files
└── scripts/                     # Utility scripts

MIT

Contributions are welcome! Feel free to submit pull requests with new words, features, or improvements.