Rather surprisingly, at least for me, the most interesting and compelling part of this flim were the prisoner/participants. Their interactions and backstories and even brief attempts to piece together some connection between them all and the experiment were all significantly more interesting than the police investigation framing narrative or the goofy doctor.
As for where it starts to come apart is the ending. After apparently proudly setting up little clues throughout the film, it indulges itself in a bit overly long expo-dump basically revealing some shocking and disturbing twist behind the motivation of the experiment, but nothing really regarding the experiment itself.
The problem with this is that the "disturbing twist" was somewhat easily figured out long, long before it unfolded. And given how the ending went, it was seemingly clear that this twist was all they had, and they fumbled it pretty badly.
Without spoiling it, I can only say, as an unrelated but parallel example; when a criminal acts like a weirdo and a freak in front of the police, they are going to attract attention to themselves. And the audience is going to be all over it.