Skip to content

Potential memory leak #144

Open
Open
Potential memory leak#144

Description

@esiqveland
esiqveland
opened on Mar 30, 2023

We are testing JVips as mainly a jpeg thumbnail processor. So far we only pass a java byte[] array, as we already hold this on the heap.

This is roughly what we do:

        byte[] contents = thumb.getBlob();
        try (VipsImage image = new VipsImage(contents, contents.length)) {
          ...
          image.colorspace(Srgb)
          image.thumbnail(...)
          image.autorot();
 
          var output = image.writeJPEGToArray(JPEG_COMPRESSION_QUALITY, true);
          return output;
       }

But when we enable the JVips processor, several instances are almost immediately going OOM:

image

Processing with the above code was enabled around 09:17 in the above.

Is this something you have seen with VipsImage or are we using it wrong?

Is there a way I can help track this down?

Activity

  1. hauf-rakuten commented on Jul 10, 2023

    @hauf-rakuten
    hauf-rakuten
    on Jul 10, 2023

    Unfortunately we have the very same situation. I could improve the situation a little bit by turning of caching and multi thread with:
    VipsContext.setMaxCache(0); VipsContext.setConcurrency(1);
    but still the memory usage continuously increases. Its not the java heap that increases but System Memory. If you shutdown Libvips context VipsContext.shutdown(); System Memory is freed up but then the context is not usable anymore.

  2. esiqveland commented on Jul 11, 2023

    @esiqveland
    esiqveland
    on Jul 11, 2023
    Author

    @hauf-rakuten have you tried holding one VipsContext per thread, like a threadlocal? Not sure if that is possible, but worth trying. I kind of gave up on this and just delegate to vips/vipsthumbnail as subprocesses

  3. hauf-rakuten commented on Jul 12, 2023

    @hauf-rakuten
    hauf-rakuten
    on Jul 12, 2023

    @esiqveland thanks for your reply. Thats the point there is no way to control the VipsContext. It is initiated on the first call of any JVips class and the only thing I found you can do is to set some flags to it and close it. But most likely I have over looked something. Thats why I am asking.

  4. frederico-apolonia commented on Apr 18, 2024

    @frederico-apolonia
    frederico-apolonia
    on Apr 18, 2024

    @hauf-rakuten were you able to find out what was causing the memory leak?

  5. hauf-rakuten commented on Apr 18, 2024

    @hauf-rakuten
    hauf-rakuten
    on Apr 18, 2024

    At the end I was able to fix it by switching the glib memory allocator that is default for debian to mimalloc and this reduced the memory usage dramatically and also solved the issue with the memory leak. There is a known issue with multithreaded apps in glib but I am not sure if thats really the cause for the issue I saw.

  6. frederico-apolonia commented on Apr 19, 2024

    @frederico-apolonia
    frederico-apolonia
    on Apr 19, 2024

    Thanks for your quick response, @hauf-rakuten . Did you just swap glib with mimalloc? Mimalloc doesn't have the g_object_free and other g_ functions that glib has, afaik

  7. hauf-rakuten commented on Apr 23, 2024

    @hauf-rakuten
    hauf-rakuten
    on Apr 23, 2024

    Yes, we simply replaced glib with mimalloc. We intensively tested and also measured emory usage even with other implementations like tcmalloc and jemalloc. Haven't seen any issue like missing functions but mimalloc turned out to perform best for us.

  8. weltonrodrigo commented on Jul 25, 2025

    @weltonrodrigo
    weltonrodrigo
    on Jul 25, 2025

    If this helps, I can see it printing errors that always grow:

    consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:50.462: read gave 8 warnings
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:50.463: VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Corrupt JPEG data: premature end of data segment
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:50.752: read gave 6 warnings
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:50.752: VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Corrupt JPEG data: premature end of data segment
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:56.440: read gave 4 warnings
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:56.440: VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Corrupt JPEG data: premature end of data segment
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:57.722: read gave 2 warnings
consumer-86c6c886b9-26vdm consumer 
consumer-86c6c886b9-26vdm consumer (process:1): VIPS-WARNING **: 19:36:57.722: VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Corrupt JPEG data: premature end of data segment
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer VipsJpeg: Premature end of JPEG file
consumer-86c6c886b9-26vdm consumer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions