Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jul 2, 2025

This PR adds a new force_mode_constant parameter to EmissionsTracker that allows users to bypass psutil completely and use constant CPU power estimation instead. This addresses significant performance overhead issues where psutil.cpu_percent(interval=0.5) calls were causing 10x+ slowdowns in ML training workflows.

Problem

Users reported severe performance overhead when tracking short ML training iterations:

  • Without CodeCarbon: 11 seconds
  • With CodeCarbon v3.0.2: 1 minute 55 seconds (10x slower!)

The overhead was caused by repeated psutil.cpu_percent(interval=0.5) calls that take 0.5 seconds each. Additionally, uninstalling psutil to force fallback mode resulted in ModuleNotFoundError due to unconditional imports.

Solution

1. Graceful psutil imports

Made psutil imports conditional in codecarbon/core/cpu.py and codecarbon/core/util.py:

try:
    import psutil
    PSUTIL_AVAILABLE = True
except ImportError:
    PSUTIL_AVAILABLE = False
    psutil = None

2. New force_mode_constant parameter

Added force_mode_constant parameter to bypass psutil entirely:

# Bypass psutil completely - eliminates 0.5s overhead per measurement
tracker = EmissionsTracker(force_mode_constant=True)

# Works with custom CPU power for better accuracy
tracker = EmissionsTracker(
    force_mode_constant=True, 
    force_cpu_power=100  # 100W TDP
)

3. Updated CPU tracking logic

Modified codecarbon/core/resource_tracker.py to prioritize force_mode_constant before all other tracking methods, ensuring it completely bypasses psutil when enabled.

Features

  • Zero psutil overhead: Completely eliminates psutil.cpu_percent() calls
  • Graceful fallback: Handles missing psutil without crashing
  • Custom power support: Works with force_cpu_power parameter
  • Precedence control: Takes priority over force_mode_cpu_load
  • Backward compatibility: No breaking changes to existing code
  • Comprehensive testing: Full test suite covering all scenarios

Performance Impact

Testing shows successful elimination of psutil overhead:

  • Normal mode: Uses CPU load tracking (slower due to psutil calls)
  • Force constant mode: Uses constant power estimation (eliminates overhead)
  • Both produce valid emissions measurements

Documentation

Added parameter documentation to docs/edit/parameters.rst:

* - force_mode_constant
  - | Force the use of constant mode for CPU power consumption measurement,
    | defaults to ``False``. When enabled, bypasses psutil completely and
    | uses a constant power consumption based on CPU TDP. Useful when
    | psutil overhead is significant or psutil is unavailable.

Fixes #878.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • get.geojs.io
    • Triggering command: python /tmp/test_force_constant.py (dns block)
    • Triggering command: python -c from codecarbon import EmissionsTracker import tempfile with tempfile.TemporaryDirectory() as temp_dir: tracker = EmissionsTracker(output_dir=temp_dir) print('Available attributes:', [attr for attr in dir(tracker) if 'resource' in attr.lower() or 'tracker' in attr.lower()]) (dns block)
    • Triggering command: python /tmp/test_simple_constant.py (dns block)
  • ip-api.com
    • Triggering command: python /tmp/test_force_constant.py (dns block)
    • Triggering command: python -c from codecarbon import EmissionsTracker import tempfile with tempfile.TemporaryDirectory() as temp_dir: tracker = EmissionsTracker(output_dir=temp_dir) print('Available attributes:', [attr for attr in dir(tracker) if 'resource' in attr.lower() or 'tracker' in attr.lower()]) (dns block)
    • Triggering command: python /tmp/test_simple_constant.py (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Unable to bypass psutil (requesting feature to force constant mode) Add force_mode_constant parameter to bypass psutil overhead Jul 2, 2025
Copilot AI requested a review from benoit-cty July 2, 2025 08:47
Copilot finished work on behalf of benoit-cty July 2, 2025 08:47
@benoit-cty benoit-cty marked this pull request as ready for review September 24, 2025 17:43
@benoit-cty
Copy link
Contributor

benoit-cty commented Sep 24, 2025

Tested working with:

CODECARBON_FORCE_MODE_CONSTANT=True uv run codecarbon monitor

@inimaz as it's me who run Copilot, I can merge by myself 😉

Copilot AI and others added 4 commits September 24, 2025 19:47
- Make psutil imports graceful in cpu.py and util.py
- Add force_mode_constant parameter to EmissionsTracker
- Update CPU tracking logic to prioritize force_mode_constant
- Add comprehensive test suite for new functionality
- Update documentation with new parameter details

Co-authored-by: benoit-cty <6603048+benoit-cty@users.noreply.github.com>
@benoit-cty benoit-cty requested a review from inimaz September 24, 2025 17:49
Copy link
Collaborator

@inimaz inimaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thanks! Minor comment

:param force_cpu_power: cpu power to be used instead of automatic detection.
:param force_ram_power: ram power to be used instead of automatic detection.
:param pue: PUE (Power Usage Effectiveness) of the datacenter.
:param force_mode_cpu_load: Force the addition of a CPU in MODE_CPU_LOAD
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to have force_mode_cpu_load=True and force_mode_constant=True at the same time? What does it mean in that case?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to bypass psutil (requesting feature to force constant mode)

3 participants