Repository navigation
How to use tqdm to log progress bars in cloudwatch logs? #966
Description
Activity
Hey, I'm also curious how to do this. I get the following ugly printout when using cloudwatch:
It looks like it doesn't support carriage return?
I think it's because of the way tqdm erases the progress bar and redraw it in place. But cloudwaatch can not go back in time.
You could use
position=-1so it only appends to the log, and then I think it should display each progress bar log in one cloudwatch entry (rather than one big blob).from time import sleep from tqdm import tqdm if __name__ == "__main__": for i in tqdm(range(9), position=-1): sleep(0.1)Then you'd get:
0%| | 0/9 [00:00<?, ?it/s] 11%|█ | 1/9 [00:00<00:00, 9.81it/s] 22%|██▏ | 2/9 [00:00<00:00, 9.76it/s] 33%|███▎ | 3/9 [00:00<00:00, 9.73it/s] 44%|████▍ | 4/9 [00:00<00:00, 9.65it/s] 56%|█████▌ | 5/9 [00:00<00:00, 9.73it/s] 67%|██████▋ | 6/9 [00:00<00:00, 9.70it/s] 78%|███████▊ | 7/9 [00:00<00:00, 9.61it/s] 89%|████████▉ | 8/9 [00:00<00:00, 9.57it/s] 100%|██████████| 9/9 [00:00<00:00, 9.63it/s]Reacted by Austin Welch, Alec Merdler, 66li, Lou Kratz, Faisal Malik and Nick AndersonReacted by Hervé Nivon@casperdcl I saw you recently added this feature: #1491
For context, when running stuff in AWS, I'm trying to have each
tqdmstatement on a different line.
Otherwise it looks like a giant blob in aws cloudwatch, and we get warnings all over the place.Previously I had to do something like this:
import os IS_AWS_BATCH: bool = "AWS_EXECUTION_ENV" in os.environ TQDM_POSITION: Optional[int] = -1 if IS_AWS_BATCH else None # .... for _ in tqdm(data, position=TQDM_POSITION): passIt works but I need to remember to import
TQDM_POSITIONand plug the position for everytqdmcall.I was hopping with your recent change I could use
export TQDM_POSITION=-1. But it doesn't work.TQDM_POSITION='-1' python -c " import tqdm from time import sleep for _ in tqdm.tqdm(range(5)): sleep(0.1) "
Gives me:
File "<string>", line 5, in <module> File "/lib/python3.10/site-packages/tqdm/std.py", line 1091, in __init__ self.pos = self._get_free_pos(self) if position is None else -position TypeError: bad operand type for unary -: 'str' Exception ignored in: <function tqdm.__del__ at 0x104fdd090> Traceback (most recent call last): File "/lib/python3.10/site-packages/tqdm/std.py", line 1147, in __del__ self.close() File "/lib/python3.10/site-packages/tqdm/std.py", line 1273, in close pos = abs(self.pos) AttributeError: 'tqdm' object has no attribute 'pos'Do you know what I'm doing wrong? And would you be able to suggest another way to achieve this?
- added 3 commits that reference this issue
on Aug 9, 2023 Why doesn't
from time import sleep from tqdm import tqdm if __name__ == "__main__": for i in tqdm(range(9), position=-1): sleep(1)print each update to the progress bar on a new line? According to the earlier posts it seems like it should?
Why doesn't
from time import sleep from tqdm import tqdm if __name__ == "__main__": for i in tqdm(range(9), position=-1): sleep(1)print each update to the progress bar on a new line? According to the earlier posts it seems like it should?
@N-Demir it depends on where you run your python program.
For example if I use the terminal, they all go on the same line.
But if I run in pycharm they all go on a different line. And if I run in AWS with cloudwatch, again, they all go on a different line
I have released a package called tqdm-loggable that automatically converts progress bars to time-interval logged messages.
When the application is running in a headless mode, progress bar are not displayed, but instead you get a log message at a regular interval during the progress bar running:
tqdm_logging.py :139 2022-09-21 12:13:56,307 Progress on:Sample progress 21.0kit/60.0kit rate:1,982.9it/s remaining:00:19 elapsed:00:10 postfix:Current time=2022-09-21 10:13:55.801787 tqdm_logging.py :139 2022-09-21 12:14:06,392 Progress on:Sample progress 41.0kit/60.0kit rate:1,984.1it/s remaining:00:09 elapsed:00:20 postfix:Current time=2022-09-21 10:14:05.890220
I want to log progress of my API in cloudwatch. Is it possible to do the same?