You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi Dev Team, thanks for the package, its one of my favs since y'all implimenation is straightforward and even include some improvements from the paper.
My issue is that recently I used the package to cluster large corpus of text (the tf-idf of the corpus). The RAM on my computer instance is 1.5 TB so there definately is room for memory.
But what tends to happen is, if I pass an array/dataframe with more that 50k observations the program shuts down (usually at the "self.kernel_size = _core.get_kernel_size(self.distances, self.fraction" line). The data sent in is all numeric and the distances are calculated using from sklearn.metrics.pairwise.cosine_similarity.
When it shuts down it gives no error message except:
"The python program has shutdown"
Do you get the same issue if you try with, for example, only 10k of your data points? If you try with the last 10k or 1k or whatever or whatever of your data?
The code for guessing a kernel size allocates a scratch array that scales with the data, which can't be helping.
I would recommending trying setting kernel_size explicitly when you construct your Cluster. If you can't think of a good value, you can always use the guess algorithm on some small subset of the data:
So technically the computer should be able to handle a distance matrix of 600k by 600k. But the program only works with a matrix of almost 50k by 50k.
It works with broken up portions of the data, but the cluster centers and cluster assignments are all relative. Thus I fear breaking up the data into multiple sets to create multiple clustering as it may impact the density and distance metrics. and make it difficult or impossible to find those smaller rarer clusters.
Especially considering even clustering with 900k observations that will account for about 1% of our data.
I'm still thinking through and researching that statistical theory behind such sampling in this use case. Particularly the number of features (words) are in the thousands.
I was not directly involved in this project, but here are my 50 cents to this topic. My suspicion is that within the C-extension we use the wrong (or lets say too small) data types for the amount and dimension of the data points. So if this number exceeds the data type horrible things will happen to the memory access and the program is terminated by the operating system.
I think we should exchange the data type for N (amount of data) and dim(ension) with size_t instead of plain (signed) int. Unfortunately I do not have the time to do this, but it should be a pretty straight forward change.
If your base your work upon mine, at single word of credit would have been nice, instead of a reference to "untrusted" PR ;)
How long did we used to work together?
I'm so sorry! You're right, that was a clumsy and unfair way to put it. Your diagnosis back in 2020 was spot on: too-small integer types for N is exactly what I ended up fixing in the end. I should've credited you directly instead of lumping your PR in with an unrelated one under a dismissive label. My fault entirely!
but it's good to hear from you --- it's been a while.
Thanks Chris, how are you doing? Good to hear from you after all this time.
--
That's the way AI puts it nowadays. The contexts are kinda limited like data centers. But it actually mentioned the previous work. It just slipped my attention.
I'm doing ok, thanks! Trying to revisit old projects and tying up loose ends while I have a bit of time on my hands :)
What have you been up to?
Oh, btw, I wanted to swap mixed_sort for qsort as you suggested (good instinct to prefer standard library over custom solution). But when I ran a comparison between those two options, mixed_sort ended up being faster (macOS/clang, -O3 -std=c99):
just a quick side note: the stdlib qsort has to dereference the comparator function pointer for every element, that's where factor 2 is originating from.
Activity
Linux-cpp-lisp commentedon Mar 26, 2020
Do you get the same issue if you try with, for example, only 10k of your data points? If you try with the last 10k or 1k or whatever or whatever of your data?
The code for guessing a kernel size allocates a scratch array that scales with the data, which can't be helping.
I would recommending trying setting
kernel_sizeexplicitly when you construct yourCluster. If you can't think of a good value, you can always use the guess algorithm on some small subset of the data:quantkeyvis commentedon Mar 26, 2020
So technically the computer should be able to handle a distance matrix of 600k by 600k. But the program only works with a matrix of almost 50k by 50k.
It works with broken up portions of the data, but the cluster centers and cluster assignments are all relative. Thus I fear breaking up the data into multiple sets to create multiple clustering as it may impact the density and distance metrics. and make it difficult or impossible to find those smaller rarer clusters.
Especially considering even clustering with 900k observations that will account for about 1% of our data.
I'm still thinking through and researching that statistical theory behind such sampling in this use case. Particularly the number of features (words) are in the thousands.
Linux-cpp-lisp commentedon Mar 26, 2020
I completely understand your reluctance to break it up; I was more wondering from a debugging point of view.
marscher commentedon Jun 4, 2020
I was not directly involved in this project, but here are my 50 cents to this topic. My suspicion is that within the C-extension we use the wrong (or lets say too small) data types for the amount and dimension of the data points. So if this number exceeds the data type horrible things will happen to the memory access and the program is terminated by the operating system.
I think we should exchange the data type for N (amount of data) and dim(ension) with size_t instead of plain (signed) int. Unfortunately I do not have the time to do this, but it should be a pretty straight forward change.
marscher commentedon Aug 13, 2020
I have a proposed fix for you problem. I'd be glad, if you can give it a try. You can install the fixing branch like this with pip:
pip install git+https://github.com/marscher/pydpc@use_long_dtype_for_indexing_and_stdlib_quicksort
Merge pull request #21 from cwehmeyer/fix-old-issues
marscher commentedon Jul 3, 2026
If your base your work upon mine, at single word of credit would have been nice, instead of a reference to "untrusted" PR ;)
How long did we used to work together?
cwehmeyer commentedon Jul 3, 2026
I'm so sorry! You're right, that was a clumsy and unfair way to put it. Your diagnosis back in 2020 was spot on: too-small integer types for N is exactly what I ended up fixing in the end. I should've credited you directly instead of lumping your PR in with an unrelated one under a dismissive label. My fault entirely!
but it's good to hear from you --- it's been a while.
marscher commentedon Jul 3, 2026
Thanks Chris, how are you doing? Good to hear from you after all this time.
--
That's the way AI puts it nowadays. The contexts are kinda limited like data centers. But it actually mentioned the previous work. It just slipped my attention.
cwehmeyer commentedon Jul 3, 2026
I'm doing ok, thanks! Trying to revisit old projects and tying up loose ends while I have a bit of time on my hands :)
What have you been up to?
Oh, btw, I wanted to swap
mixed_sortforqsortas you suggested (good instinct to prefer standard library over custom solution). But when I ran a comparison between those two options,mixed_sortended up being faster (macOS/clang, -O3 -std=c99):I did not expect that, tbh... Guess I'll keep
mixed_sortfor the time being. Thanks a lot for the suggestion, though!marscher commentedon Jul 3, 2026
Thanks for the benchmark. I should have done that back in the day, when I proposed it.
marscher commentedon Jul 3, 2026
Glad to hear you're fine. I'm still applying for jobs atm 👍
marscher commentedon Jul 3, 2026
just a quick side note: the stdlib qsort has to dereference the comparator function pointer for every element, that's where factor 2 is originating from.