An automated system that uses the Brave Search API to identify trending products for dropshipping.

Unlike a traditional web crawler, this system uses the Brave Search API to discover potential products by performing targeted searches. The process takes place in three main steps:

flowchart LR
    A["Discovery\nBrave Search API"] --> B["Analysis\nScoring Algorithm"]
    B --> C["Decision\nIndex / Watch / Skip"]
    C --> D["Integration\nE-commerce Platform"]
    style A fill:#3498db,stroke:#333,stroke-width:2px
    style B fill:#2ecc71,stroke:#333,stroke-width:2px
    style C fill:#e74c3c,stroke:#333,stroke-width:2px
    style D fill:#9b59b6,stroke:#333,stroke-width:2px

1. Discovery: Predefined queries are sent to the Brave Search API to find potential products
2. Analysis: Each product is evaluated according to popularity, profitability, competition and seasonality
3. Decision: Based on this analysis, the system recommends adding, monitoring or ignoring each product

Learn more about how it works in detail

flowchart TD
    API["Brave Search API"] --> Crawler
    AliExpress["AliExpress API"] --> ProductSource
    
    subgraph Core["Core System"]
        Crawler --> DB[(MongoDB)]
        ProductSource["Product Source"] --> DB
        DB --> Analyzer
        Analyzer --> DB
        DB --> CatalogManager["Catalog Manager"]
    end
    
    subgraph Integrations["E-commerce Integrations"]
        CatalogManager --> Shopify["Shopify Store"]
        CatalogManager --> WooCommerce["WooCommerce Store"]
        CatalogManager --> OtherPlatforms["Other Platforms"]
    end
    
    DB --> WebUI["Web Dashboard"]
    User[User] --> WebUI
    WebUI --> IntegrationsUI["Integrations UI"]
    IntegrationsUI --> Integrations
    
    classDef external fill:#f9f9f9,stroke:#333,stroke-width:1px
    classDef core fill:#d6eaf8,stroke:#333,stroke-width:2px
    classDef ui fill:#e8f8f5,stroke:#333,stroke-width:2px
    classDef integrations fill:#f8d9e9,stroke:#333,stroke-width:2px
    
    class API,AliExpress external
    class Core core
    class WebUI,User,IntegrationsUI ui
    class Integrations,Shopify,WooCommerce,OtherPlatforms integrations

This project aims to develop a web application that:

1. Automatically crawls the web at regular intervals (every hour)
2. Identifies trending products on various platforms
3. Analyzes their potential for dropshipping
4. Generates recommendations for adding or removing products
5. Automates the indexing/deindexing of products in your catalog

• Backend: Node.js with Express
• Frontend: HTML, CSS, JavaScript
• Database: MongoDB
• Crawling: Brave Search API
• Integrations: Shopify, WooCommerce, AliExpress
• Deployment: Docker and GitHub Actions

• Node.js 14+ and npm installed
• MongoDB installed and running (optional - the system can run in demo mode without a database)
• Brave Search API key (optional - the system uses test data if no key is configured)
• For e-commerce integrations: API credentials for your platform (Shopify, WooCommerce, etc.)

MongoDB Installation Guide

# Clone the repository
git clone https://github.com/nabz0r/dropship-trend-crawler.git
cd dropship-trend-crawler

# Install dependencies
npm install

# Configuration
cp .env.example .env
# Edit the .env file with your API keys and other settings

npm run dev

npm start

The application will be available at http://localhost:3000 (or the port specified in your .env file).

The system can run without MongoDB and Brave Search API keys using mock data. This is great for quickly testing features without a full setup.

You can customize the system behavior via the config/crawler-settings.json file or via the web interface:

• Modify search queries
• Adjust the weights of the different analysis factors
• Configure decision thresholds
• Enable/disable automatic indexing

pie title Product Scoring Factors
    "Popularity" : 40
    "Profitability" : 30
    "Competition" : 20
    "Seasonality" : 10

The system now includes built-in integrations with popular e-commerce platforms:

flowchart LR
    System["DropShip\nCrawler"] -- "Product Data" --> Integrations

    subgraph Integrations["Platform Integrations"]
        Shopify["Shopify\nIntegration"]
        WooCommerce["WooCommerce\nIntegration"] 
        AliExpress["AliExpress\nIntegration"]
    end

    Shopify -- "API Calls" --> ShopifyStore["Your Shopify\nStore"]
    WooCommerce -- "API Calls" --> WooStore["Your WooCommerce\nStore"]
    AliExpress -- "Product Research" --> AliData["Sourcing\nData"]

    style System fill:#3498db,stroke:#333,stroke-width:2px
    style Integrations fill:#f8d9e9,stroke:#333,stroke-width:1px
    style ShopifyStore fill:#95c675,stroke:#333,stroke-width:2px
    style WooStore fill:#96588a,stroke:#333,stroke-width:2px
    style AliData fill:#e74c3c,stroke:#333,stroke-width:2px

Configure your platform credentials in the dedicated Integrations page and enable automatic product publishing

The REST API exposes the following endpoints:

To integrate this system with your dropshipping platform, you can:

1. Use the built-in integrations for Shopify and WooCommerce
2. Modify the addProductToCatalog and removeProductFromCatalog methods in src/services/catalogManager.js to connect to other platforms
3. Develop specific middlewares to translate data between our format and your platform's.
4. Use the REST API to develop your own user interface.

See the task list in the TASKS.md file for features planned for future releases.

See the docs/ folder for detailed information on:

• How the system works
• Using the Brave Search API
• System architecture
• Installing MongoDB

We welcome:

• 💻 Network Engineers
• 👀 Research Scientists
• 🌎 Cloud Architects
• 🤖 AI/ML Specialists

MIT License - Innovation without Boundaries