Skip to content

Package shuts down on large data. #6

Closed

Description

@quantkeyvis

Hi Dev Team, thanks for the package, its one of my favs since y'all implimenation is straightforward and even include some improvements from the paper.

My issue is that recently I used the package to cluster large corpus of text (the tf-idf of the corpus). The RAM on my computer instance is 1.5 TB so there definately is room for memory.

But what tends to happen is, if I pass an array/dataframe with more that 50k observations the program shuts down (usually at the "self.kernel_size = _core.get_kernel_size(self.distances, self.fraction" line). The data sent in is all numeric and the distances are calculated using from sklearn.metrics.pairwise.cosine_similarity.

When it shuts down it gives no error message except:
"The python program has shutdown"

Activity

  1. Linux-cpp-lisp commented on Mar 26, 2020

    @Linux-cpp-lisp
    Contributor

    Do you get the same issue if you try with, for example, only 10k of your data points? If you try with the last 10k or 1k or whatever or whatever of your data?

    The code for guessing a kernel size allocates a scratch array that scales with the data, which can't be helping.

    I would recommending trying setting kernel_size explicitly when you construct your Cluster. If you can't think of a good value, you can always use the guess algorithm on some small subset of the data:

    guess_kern_size = _core.get_kernel_size(dist_matrix[:500, :500], fraction)
    # ...
    Cluster(
        kernel_size = guess_kern_size,
        fraction = fraction,
        # etc.
    )
    
  2. quantkeyvis commented on Mar 26, 2020

    @quantkeyvis
    Author

    So technically the computer should be able to handle a distance matrix of 600k by 600k. But the program only works with a matrix of almost 50k by 50k.

    It works with broken up portions of the data, but the cluster centers and cluster assignments are all relative. Thus I fear breaking up the data into multiple sets to create multiple clustering as it may impact the density and distance metrics. and make it difficult or impossible to find those smaller rarer clusters.

    Especially considering even clustering with 900k observations that will account for about 1% of our data.

    I'm still thinking through and researching that statistical theory behind such sampling in this use case. Particularly the number of features (words) are in the thousands.

  3. Linux-cpp-lisp commented on Mar 26, 2020

    @Linux-cpp-lisp
    Contributor

    I completely understand your reluctance to break it up; I was more wondering from a debugging point of view.

  4. marscher commented on Jun 4, 2020

    @marscher

    I was not directly involved in this project, but here are my 50 cents to this topic. My suspicion is that within the C-extension we use the wrong (or lets say too small) data types for the amount and dimension of the data points. So if this number exceeds the data type horrible things will happen to the memory access and the program is terminated by the operating system.
    I think we should exchange the data type for N (amount of data) and dim(ension) with size_t instead of plain (signed) int. Unfortunately I do not have the time to do this, but it should be a pretty straight forward change.

  5. marscher commented on Aug 13, 2020

    @marscher

    I have a proposed fix for you problem. I'd be glad, if you can give it a try. You can install the fixing branch like this with pip:

    pip install git+https://github.com/marscher/pydpc@use_long_dtype_for_indexing_and_stdlib_quicksort

  6. added a commit that references this issue on Jul 2, 2026
  7. marscher commented on Jul 3, 2026

    @marscher

    If your base your work upon mine, at single word of credit would have been nice, instead of a reference to "untrusted" PR ;)
    How long did we used to work together?

  8. cwehmeyer commented on Jul 3, 2026

    @cwehmeyer
    Owner

    I'm so sorry! You're right, that was a clumsy and unfair way to put it. Your diagnosis back in 2020 was spot on: too-small integer types for N is exactly what I ended up fixing in the end. I should've credited you directly instead of lumping your PR in with an unrelated one under a dismissive label. My fault entirely!

    but it's good to hear from you --- it's been a while.

  9. marscher commented on Jul 3, 2026

    @marscher

    Thanks Chris, how are you doing? Good to hear from you after all this time.

    --
    That's the way AI puts it nowadays. The contexts are kinda limited like data centers. But it actually mentioned the previous work. It just slipped my attention.

  10. cwehmeyer commented on Jul 3, 2026

    @cwehmeyer
    Owner

    I'm doing ok, thanks! Trying to revisit old projects and tying up loose ends while I have a bit of time on my hands :)

    What have you been up to?

    Oh, btw, I wanted to swap mixed_sort for qsort as you suggested (good instinct to prefer standard library over custom solution). But when I ran a comparison between those two options, mixed_sort ended up being faster (macOS/clang, -O3 -std=c99):

       npoints        n_pairs   mixed_sort        qsort speedup(x)
           200          19900     0.000986     0.002097       2.13
          1000         499500     0.034383     0.070056       2.04
          2000        1999000     0.162962     0.321099       1.97
          5000       12497500     1.120215     2.146552       1.92
         10000       49995000     4.805263     9.453281       1.97
    

    I did not expect that, tbh... Guess I'll keep mixed_sort for the time being. Thanks a lot for the suggestion, though!

  11. marscher commented on Jul 3, 2026

    @marscher

    Thanks for the benchmark. I should have done that back in the day, when I proposed it.

  12. marscher commented on Jul 3, 2026

    @marscher

    Glad to hear you're fine. I'm still applying for jobs atm 👍

  13. marscher commented on Jul 3, 2026

    @marscher

    just a quick side note: the stdlib qsort has to dereference the comparator function pointer for every element, that's where factor 2 is originating from.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions