Update to Cog v0.16.8 and vllm 0.11 #41
Conversation
- Update vllm from 0.5.3.post1 to 0.11.0 - Update Cog from 0.10.0a18 to 0.16.8 - Replace ConcatenateIterator with AsyncConcatenateIterator (introduced in Cog 0.14.0) - Remove unnecessary sed patch for vllm weight_utils.py (no longer needed) - Update pyproject.toml to use Cog >= 0.16.8 - No pydantic version constraints, allowing 2.0+ compatibility 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Pull Request Overview
This PR updates the project to use Cog v0.16.8 and vLLM 0.11.0, replacing the previous alpha versions (Cog v0.10.0a10/a18 and vLLM 0.5.3.post1). The update includes API compatibility changes, improved mock infrastructure for testing, and updated documentation.
Key changes:
- Migration from
ConcatenateIteratortoAsyncConcatenateIteratorfor async streaming - Enhanced handling of URLFile objects from newer Cog versions with regex-based URL extraction
- Test infrastructure improvements with expanded mocking for Cog components
- Parameter name standardization (
max_new_tokens→max_tokens)
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cog.yaml | Updates CUDA version to 12.4, vLLM to 0.11.0, Cog to 0.16.8, and pget to 0.11.0 |
| pyproject.toml | Updates Cog dependency constraint and adds pytest asyncio configuration |
| predict.py | Implements AsyncConcatenateIterator, adds URLFile handling, updates tokenizer initialization, and adds safety checks for optional Cog features |
| test.sh | Renames parameter from max_new_tokens to max_tokens |
| tests/unit/test_predict.py | Enhances mock setup with additional Cog components before imports |
| train.py | Adds pylint disable comment for positional arguments warning |
| README.md | Restructures documentation with clearer installation and usage instructions |
| .python-version | Removes Python version pin file |
| .dockerignore | Adds .python-version to ignored files |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
predict.py
Outdated
| # Handle URLFile objects from Cog | ||
| if hasattr(weights, '__str__'): | ||
| # URLFile objects need special handling | ||
| weights_str = str(weights) | ||
| # Check if it's a URLFile representation (contains 'URLFile at') | ||
| if 'URLFile' in weights_str: | ||
| # Extract the actual URL from the representation | ||
| # Format: "<URLFile at 0x... for 'URL'>" | ||
| match = re.search(r"for '([^']+)'", weights_str) | ||
| if match: | ||
| weights_path = match.group(1) | ||
| else: | ||
| raise ValueError(f"Could not extract URL from URLFile: {weights_str}") | ||
| else: | ||
| weights_path = weights_str |
There was a problem hiding this comment.
The URLFile handling logic relies on fragile string parsing of the object's string representation. This approach is brittle and could break if the URLFile's __repr__ format changes. Consider checking if the weights object has a .url or .path attribute to access the URL directly, or use isinstance() checks with the actual URLFile class if available.
| # Handle URLFile objects from Cog | |
| if hasattr(weights, '__str__'): | |
| # URLFile objects need special handling | |
| weights_str = str(weights) | |
| # Check if it's a URLFile representation (contains 'URLFile at') | |
| if 'URLFile' in weights_str: | |
| # Extract the actual URL from the representation | |
| # Format: "<URLFile at 0x... for 'URL'>" | |
| match = re.search(r"for '([^']+)'", weights_str) | |
| if match: | |
| weights_path = match.group(1) | |
| else: | |
| raise ValueError(f"Could not extract URL from URLFile: {weights_str}") | |
| else: | |
| weights_path = weights_str | |
| # Handle URLFile objects from Cog robustly | |
| if hasattr(weights, "url"): | |
| weights_path = weights.url | |
| elif hasattr(weights, "path"): | |
| weights_path = weights.path |
| if hasattr(self, 'log'): | ||
| self.log( | ||
| "Warning: ignoring system prompt because no chat template was configured" | ||
| ) |
There was a problem hiding this comment.
The conditional check if hasattr(self, 'log') is repeated multiple times (lines 283, 331, and 339). Consider extracting this into a helper method like safe_log() to reduce duplication and improve maintainability.
README.md
Outdated
| ## Local Development | ||
| You'll need: | ||
| - A URL to a `.tar` archive containing your model weights, or a local path to model files | ||
| - Cog installed ([v0.14.0](https://github.com/replicate/cog/releases/tag/v0.14.0) or later) |
There was a problem hiding this comment.
The documentation states Cog v0.14.0 or later is required, but the actual installation command and cog.yaml specify v0.16.8. This inconsistency could confuse users. Consider updating the prerequisite to specify v0.16.8 explicitly to match the installation instructions.
| - Cog installed ([v0.14.0](https://github.com/replicate/cog/releases/tag/v0.14.0) or later) | |
| - Cog installed ([v0.16.8](https://github.com/replicate/cog/releases/tag/v0.16.8)) |
Update Cog & vllm version and get it to a working state where test.sh runs