[Bug]: git operations very slow and time out #35644
Comments
|
cc: @nidhi-nair - is it possible to debug with the info provided? |
|
From the timestamps in the logs, it looks like the git reset and git save operations are taking a really long time. The lock acquisition went through so this is not an issue with parallel usage, I'm inclined to think that the application has grown to be that large over time. @naptha77 would you be able to share the size of your repository? I am also sharing a command for total number of files within. Please execute the following inside the repository directory: |
|
@nidhi-nair Those commands returned 1.4G and 41451 My actual repo is only 673 KB according to github. I cloned locally and it says there are 565 files, total 1.65 MB So I'm a bit confused - the repo is very small, but the container has some large files in it. I would have thought they'd all be ignored and the git commands would still be quick. These are the largest files in my container. Can any of them be purged to speed up git operations? Git operations have always taken a long time in appsmith since I started with it. 1.2G /appsmith-stacks/data/mongodb/collection-10--5793211097493342907.wt |
|
I created a brand new appsmith container and imported the app from my same git repo. It was still very slow but worked that time. My mongodb file is only 3.9MB in the new container: My app displays lots of images, so mongo may be caching them, slowing down the git operations over time. Although I expect it'd be ignoring *.wt files so it shouldn't affect git. The root issue still persists however which is very slow git operations. I assume it'll eventually timeout on this container as well. What range of speeds should I be expecting for git operations? Should I try using an external mongodb instance so I don't have that 1.2GB database sitting inside my container? New git operation logs showing new speeds: [2024-08-13 22:07:41,472] ... - Going to get branch list for application 66bbd7f044507f7b3f6a5de5 |
|
I should have mentioned the steps more clearly in my previous comment. I meant to check the size of the repo itself, which is 1.65MB as you mentioned. That's not too high for us to see this kind of latency. The rest of the data should not matter, those are to do with the DB, logs, and other artefacts for Appsmith. Next we can attempt to set up some observability on your system to check what step exactly is failing here. I'm just heading out for a small vacation, but would you be available for a call early next week and may be we attempt to resolve this in sync, @naptha77 ? |
|
Thanks @nidhi-nair, sounds great. The operation that is timing out is http://localhost/api/v1/git/status/app/66___37?compareRemote=true. It fails after exactly 2 minutes. Is there any config where I can increase that timeout? At least then I can keep working in the meantime. I've tried creating a new container on a new VM, importing my app then pointing it at a new bitbucket repo (as opposed to github). I still have thew same issue - the compareRemote call above takes 1.9 minutes so almost times out. So I assume it's something related to my appsmith app itself. I imported a very early version of my app into a new repo and the compareRemote took 28 seconds. But the repo is only 100k, so only 15% of the size of the new one. Anyway, looking forward to chatting next week |
|
Just wanted to say that I'm seeing the same types of delays with git operations. This has been happening from the start. It is so bad that I'm hesitant to recommend Appsmith to other developers due to this performance issue. Self hosted on Azure Container with the repo in Azure Dev Ops. |
|
@rmmlopes - Apologies for the current experience, we're aware of the poor performance, especially with Azure File System. We're working on a large project to drastically improve this, I'll share a GitHub ticket once we're closer to development. |
|
@naptha77 here's my Calendly link, please feel free to pick a slot that works for you and we'll figure out how to bring in some more visibility for you. Also, before we get into the call, please make sure to acquire access to all the components of Appsmith so the debugging proceeds seamlessly. @rmmlopes sorry that you're facing this as well. Azure FS like Akhil mentioned seems to have latency problems of its own that renders it impossible for use. I'd love to chat if you're up for it though, it would be interesting to see if there's any peculiarities in your system different from the ones we're already keeping track of. |
I am also starting to see a lot of errors on the UI when editing JS code in the appsmith UI saying something to the effect that "X function already exists" when I'm editing that piece of code. At this point I have come to the conclusion that Appsmith should not be run in Azure. I think you should update your web documentation to this effect so that people don't waste a bunch of time trying to get it to work. |
|
I decided to remove the git integration and just export the project JSON after each change, checking it into my own github repo. It means deployments are instant and always work. I may revisit later if bugs are fixed as above, but at the moment it's not worth spending any more time on |
Description
I have a self hosted appsmith instance with docker. Git integration has been working for weeks but now fails when I try to commit changes.
The git operation http://localhost/api/v1/git/status/app/668de___c6837?compareRemote=true times out after 2 minutes. The operation before that, http://localhost/api/v1/git/auto-commit/app/668___837?branchName=main works but takes 70 seconds to run.
This was working a week ago and nothing has changed. I had exactly the same issue before - it would work for weeks, then intermittently fail, then eventually fail every time. My solution was to build a new docker image and then import my project, which I'd rather not do obviously.
It may be related to having the appsmith editor open in multiple browser tabs at the same time. I was careful not to do that with this new docker image, but did it before I was having the issues.
On the previous version of appsmith it said it failed to acquire a git lock, whereas now it just times out
My host is a Windows VM which has 20GB of RAM and 16 vCPU's. I've never had any other performance issues or timeouts - it's just related to git which has always been slow.
Relevant logs below.
2024-08-13 08:00:54 backend stdout | [2024-08-13 00:00:54,165] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - Git command status is trying to acquire the lock for application id 668de550f3f42c094b3c6837
2024-08-13 08:01:00 backend stdout | [2024-08-13 00:01:00,197] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - Execute time: JGIT git_fetch, appId 668de550f3f42c094b3c6837, Time elapsed: 6023ms
2024-08-13 08:01:27 backend stdout | [2024-08-13 00:01:27,351] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - Execute time: JGIT git_reset, appId 668de550f3f42c094b3c6837, Time elapsed: 33122ms
2024-08-13 08:01:27 backend stdout | [2024-08-13 00:01:27,455] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - boundedElastic-30: Switching to the branch main
2024-08-13 08:01:58 backend stdout | [2024-08-13 00:01:58,169] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - Execute time: JGIT git_reset, appId 668de550f3f42c094b3c6837, Time elapsed: 30665ms
2024-08-13 08:02:13 backend stdout | [2024-08-13 00:02:13,836] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - Execute time: FS application save, Time elapsed: 79661ms
2024-08-13 08:02:13 backend stdout | [2024-08-13 00:02:13,933] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - boundedElastic-30: Get status for repo /appsmith-stacks/git-storage/668de202f3f42c094b3c6769/668de550f3f42c094b3c6837/amazon-appsmith, branch main
2024-08-13 08:03:11 backend stdout | [2024-08-13 00:03:11,202] requestId=51e7f43d-4784-4479-9154-80bcfdbdcbf0 userEmail=mic_____11@gmail.com traceId= spanId= - Execute time: JGIT git_reset, appId 668de550f3f42c094b3c6837, Time elapsed: 40090ms
Steps To Reproduce
Public Sample App
No response
Environment
Production
Severity
High (Blocker to building or releasing)
Issue video log
https://www.loom.com/share/329acf49732349e5beb854914f3c3e8c?sid=8daadb90-84b2-41c5-932c-1e28023da9fb
Version
Self hosted 1.30
The text was updated successfully, but these errors were encountered: