-
Notifications
You must be signed in to change notification settings - Fork 31
feature(examples): Add JBoard connector #459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,104 @@ | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Jboard API Connector Example | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| ## Connector overview | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| This connector syncs employers, job categories, and alert subscriptions data from Jboard API to your destination warehouse. The connector fetches employer profiles with their metadata, job categories with hierarchical structure, and user alert subscriptions with search criteria. It supports incremental synchronization using timestamp-based cursors and handles API rate limiting with automatic retry logic. The connector uses memory-efficient streaming to process large datasets without accumulation issues. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| ## Requirements | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - [Supported Python versions](https://github.com/fivetran/fivetran_connector_sdk/blob/main/README.md#requirements): **3.9-3.13** | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Operating system: | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Windows: 10 or later (64-bit only) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| ## Getting started | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| ## Features | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Syncs employer data, job categories, and alert subscriptions from Jboard API | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Bearer token authentication with secure API key handling (refer to `execute_api_request` function) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Page-based pagination with automatic page traversal (refer to `get_employers`, `get_categories`, and `get_alert_subscriptions` functions) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Memory-efficient streaming prevents data accumulation for large datasets using generator patterns | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Incremental synchronization using timestamp-based cursors (refer to `get_time_range` function) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Comprehensive error handling with exponential backoff retry logic (refer to `__handle_rate_limit` and `__handle_request_error` functions) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Configurable sync parameters including page size, timeout, and retry attempts | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - Cognitive complexity optimization with helper function extraction for maintainability | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| ## Configuration file | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| ```json | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| { | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "api_key": "<YOUR_JBOARD_API_KEY>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "sync_frequency_hours": "<YOUR_JBOARD_API_SYNC_FREQUENCY_HOURS>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "initial_sync_days": "<YOUR_JBOARD_API_INITIAL_SYNC_DAYS>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "max_records_per_page": "<YOUR_JBOARD_API_MAX_RECORDS_PER_PAGE>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "request_timeout_seconds": "<YOUR_JBOARD_API_REQUEST_TIMEOUT_SECONDS>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "retry_attempts": "<YOUR_JBOARD_API_RETRY_ATTEMPTS>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "enable_incremental_sync": "<YOUR_JBOARD_API_ENABLE_INCREMENTAL_SYNC>", | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| "enable_debug_logging": "<YOUR_JBOARD_API_ENABLE_DEBUG_LOGGING>" | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+26
to
+38
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| ### Configuration parameters | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `api_key`: Your Jboard API authentication key (required) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `sync_frequency_hours`: How often to sync data in hours (default: 4) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `initial_sync_days`: Days of historical data to fetch on first sync (default: 90) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `max_records_per_page`: Maximum records per API request (default: 100, range: 1-1000) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `request_timeout_seconds`: API request timeout in seconds (default: 30) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `retry_attempts`: Number of retry attempts for failed requests (default: 3) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `enable_incremental_sync`: Enable timestamp-based incremental sync (default: true) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| - `enable_debug_logging`: Enable detailed debug logs (default: false) | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+30
to
+48
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
| "sync_frequency_hours": "<YOUR_JBOARD_API_SYNC_FREQUENCY_HOURS>", | |
| "initial_sync_days": "<YOUR_JBOARD_API_INITIAL_SYNC_DAYS>", | |
| "max_records_per_page": "<YOUR_JBOARD_API_MAX_RECORDS_PER_PAGE>", | |
| "request_timeout_seconds": "<YOUR_JBOARD_API_REQUEST_TIMEOUT_SECONDS>", | |
| "retry_attempts": "<YOUR_JBOARD_API_RETRY_ATTEMPTS>", | |
| "enable_incremental_sync": "<YOUR_JBOARD_API_ENABLE_INCREMENTAL_SYNC>", | |
| "enable_debug_logging": "<YOUR_JBOARD_API_ENABLE_DEBUG_LOGGING>" | |
| } | |
| ``` | |
| ### Configuration parameters | |
| - `api_key`: Your Jboard API authentication key (required) | |
| - `sync_frequency_hours`: How often to sync data in hours (default: 4) | |
| - `initial_sync_days`: Days of historical data to fetch on first sync (default: 90) | |
| - `max_records_per_page`: Maximum records per API request (default: 100, range: 1-1000) | |
| - `request_timeout_seconds`: API request timeout in seconds (default: 30) | |
| - `retry_attempts`: Number of retry attempts for failed requests (default: 3) | |
| - `enable_incremental_sync`: Enable timestamp-based incremental sync (default: true) | |
| - `enable_debug_logging`: Enable detailed debug logs (default: false) | |
| "initial_sync_days": "<YOUR_JBOARD_API_INITIAL_SYNC_DAYS>", | |
| "max_records_per_page": "<YOUR_JBOARD_API_MAX_RECORDS_PER_PAGE>", | |
| "request_timeout_seconds": "<YOUR_JBOARD_API_REQUEST_TIMEOUT_SECONDS>", | |
| "retry_attempts": "<YOUR_JBOARD_API_RETRY_ATTEMPTS>", | |
| "enable_incremental_sync": "<YOUR_JBOARD_API_ENABLE_INCREMENTAL_SYNC>" | |
| } |
Configuration parameters
api_key: Your Jboard API authentication key (required)initial_sync_days: Days of historical data to fetch on first sync (default: 90)max_records_per_page: Maximum records per API request (default: 100, range: 1-1000)request_timeout_seconds: API request timeout in seconds (default: 30)retry_attempts: Number of retry attempts for failed requests (default: 3)enable_incremental_sync: Enable timestamp-based incremental sync (default: true)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Employer, category, and alert subscription data is mapped from Jboard API's format to normalized database columns (refer to the `__map_employer_data`, `__map_category_data`, and `__map_alert_subscription_data` functions). Nested objects like tags arrays are serialized to JSON strings, and all timestamps are converted to UTC format for consistency. | |
| Employer, category, and alert subscription data is mapped from Jboard API's format to normalized database columns (refer to the `__map_employer_data`, `__map_category_data`, and `__map_alert_subscription_data` functions). Nested objects like `tags` arrays are serialized to JSON strings, and all timestamps are converted to UTC format for consistency. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| **EMPLOYERS**: `id`, `name`, `description`, `website`, `logo_url`, `featured`, `source`, `created_at`, `updated_at`, `have_posted_jobs`, `have_a_logo`, `sync_timestamp` | |
| **CATEGORIES**: `id`, `name`, `description`, `parent_id`, `sort_order`, `is_active`, `created_at`, `updated_at`, `sync_timestamp` | |
| **ALERT_SUBSCRIPTIONS**: `id`, `email`, `query`, `location`, `search_radius`, `remote_work_only`, `category_id`, `job_type`, `tags`, `is_active`, `created_at`, `updated_at`, `sync_timestamp` | |
| EMPLOYERS: `id`, `name`, `description`, `website`, `logo_url`, `featured`, `source`, `created_at`, `updated_at`, `have_posted_jobs`, `have_a_logo`, `sync_timestamp` | |
| CATEGORIES: `id`, `name`, `description`, `parent_id`, `sort_order`, `is_active`, `created_at`, `updated_at`, `sync_timestamp` | |
| ALERT_SUBSCRIPTIONS: `id`, `email`, `query`, `location`, `search_radius`, `remote_work_only`, `category_id`, `job_type`, `tags`, `is_active`, `created_at`, `updated_at`, `sync_timestamp` |
Copilot
AI
Nov 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reference to non-existent requirements.txt file. Line 98 mentions "requirements.txt – Python dependency specification for Jboard API integration and connector requirements including faker for mock testing", but no requirements.txt file exists in the connector directory.
Either:
- Remove this reference from the "Additional files" section if no requirements.txt is needed
- Add the requirements.txt file if faker or other dependencies are actually needed for testing
Based on the connector code, no additional dependencies are needed, so this reference should be removed.
| - `requirements.txt` – Python dependency specification for Jboard API integration and connector requirements including faker for mock testing. |
Copilot
AI
Nov 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incorrect statement about requirements.txt. The README states "This connector does not require any additional packages" (line 51), but line 98 mentions "requirements.txt – Python dependency specification for Jboard API integration and connector requirements including faker for mock testing."
These statements contradict each other. If there's a requirements.txt file with faker, then the connector does have additional dependencies. Clarify which is correct and update accordingly.
| ## Additional files | |
| The connector includes several additional files to support functionality, testing, and deployment: | |
| - `requirements.txt` – Python dependency specification for Jboard API integration and connector requirements including faker for mock testing. | |
| - `configuration.json` – Configuration template for API credentials and connector parameters (should be excluded from version control). | |
| ## Additional considerations | |
| ## Requirements file | |
| The connector requires the `faker` package for mock testing and development purposes. | |
faker
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
## Additional files
The connector includes several additional files to support functionality, testing, and deployment:
- `requirements.txt` – Python dependency specification for mock testing and development.
- `configuration.json` – Configuration template for API credentials and connector parameters (should be excluded from version control).
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,10 @@ | ||||||||
| { | ||||||||
| "api_key": "<YOUR_JBOARD_API_KEY>", | ||||||||
| "sync_frequency_hours": "<YOUR_JBOARD_API_SYNC_FREQUENCY_HOURS>", | ||||||||
|
||||||||
| "sync_frequency_hours": "<YOUR_JBOARD_API_SYNC_FREQUENCY_HOURS>", |
Copilot
AI
Nov 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Placeholder descriptions are too verbose and redundant. According to the coding guidelines, placeholders should be clear and concise, describing what the user needs to provide without excessive prefixes.
Change:
<YOUR_JBOARD_API_SYNC_FREQUENCY_HOURS>→<SYNC_FREQUENCY_IN_HOURS><YOUR_JBOARD_API_INITIAL_SYNC_DAYS>→<INITIAL_SYNC_DAYS><YOUR_JBOARD_API_MAX_RECORDS_PER_PAGE>→<MAX_RECORDS_PER_PAGE><YOUR_JBOARD_API_REQUEST_TIMEOUT_SECONDS>→<REQUEST_TIMEOUT_SECONDS><YOUR_JBOARD_API_RETRY_ATTEMPTS>→<RETRY_ATTEMPTS><YOUR_JBOARD_API_ENABLE_INCREMENTAL_SYNC>→<TRUE_OR_FALSE><YOUR_JBOARD_API_ENABLE_DEBUG_LOGGING>→<TRUE_OR_FALSE>
The excessive "YOUR_JBOARD_API_" prefix doesn't add value and makes the placeholders harder to read.
Copilot
AI
Nov 20, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Configuration parameter enable_debug_logging is defined in configuration.json but never used in the connector code. Either remove this parameter from the configuration file if it's not needed, or implement debug logging logic in the connector.
| "enable_incremental_sync": "<YOUR_JBOARD_API_ENABLE_INCREMENTAL_SYNC>", | |
| "enable_debug_logging": "<YOUR_JBOARD_API_ENABLE_DEBUG_LOGGING>" | |
| "enable_incremental_sync": "<YOUR_JBOARD_API_ENABLE_INCREMENTAL_SYNC>" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.