Skip to content

Resource exhausted: OOM when allocating tensor  #10

Description

@skeletonli

Dr. Wang, thank you so much for your wonderful work.
When I run last step : python main.py, an error occored:

2019-07-15 17:38:14.500279: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:673 : Resource exhausted: OOM when allocating tensor with shape[442410,128,9,1] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

My gpu information is :
2019-07-15 17:37:54.354919: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:05:00.0
totalMemory: 11.00GiB freeMemory: 9.11GiB
2019-07-15 17:37:54.538331: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:09:00.0
totalMemory: 11.00GiB freeMemory: 9.11GiB

other information before the error occur is :
2019-07-15 17:38:14.475725: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 2265139200 totalling 2.11GiB
2019-07-15 17:38:14.480071: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 5.72GiB
2019-07-15 17:38:14.484854: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
Limit: 9244818801
InUse: 6142976256
MaxInUse: 6369439488
NumAllocs: 16017
MaxAllocSize: 2265139200

I wonder how can I solve the problem,thank you

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions