Skip to content

Db2-Plugin #1131

Open
GergesBernaba1 wants to merge 9 commits into
dbgate:masterfrom
GergesBernaba1:them-desgin-refactor
Open

Db2-Plugin #1131
GergesBernaba1 wants to merge 9 commits into
dbgate:masterfrom
GergesBernaba1:them-desgin-refactor

Conversation

@GergesBernaba1

Copy link
Copy Markdown
  • Added getStructureFixed.js to provide an enhanced implementation of the getStructure method for the DB2 driver, including error handling and a fallback to a basic implementation.
  • Updated sql.js to exclude system schemas from schema retrieval.
  • Enhanced driver.js with improved column properties and schema handling configurations.
  • Modified test-db2-driver.js to use environment variables for database connection details.
  • Introduced test-network-debug.js for debugging network requests and testing API endpoints directly.

- Implemented DB2 database connection and query execution.
- Added README.md for plugin installation and features.
- Created icon.svg for DB2 representation.
- Configured package.json with dependencies and scripts.
- Developed backend driver for DB2 with methods for connection, querying, and transaction management.
- Established frontend driver with connection fields and support for various database objects.
- Implemented SQL helper functions for table creation, dropping, and selection.
- Configured Webpack for both backend and frontend builds.
- Added Prettier configuration for code formatting.
…g, and implement schema/table/column fetching
- Added TypeScript as a development dependency in package.json.
- Created comprehensive documentation for DB2 plugin fixes, including API endpoints and testing instructions.
- Improved error handling and connection stability in the DB2 driver.
- Implemented multiple query approaches to address SQL syntax issues across different DB2 versions.
- Added test scripts for verifying API endpoint functionality and connection handling.
…nism

- Added getStructureFixed.js to provide an enhanced implementation of the getStructure method for the DB2 driver, including error handling and a fallback to a basic implementation.
- Updated sql.js to exclude system schemas from schema retrieval.
- Enhanced driver.js with improved column properties and schema handling configurations.
- Modified test-db2-driver.js to use environment variables for database connection details.
- Introduced test-network-debug.js for debugging network requests and testing API endpoints directly.
@GergesBernaba1 GergesBernaba1 changed the title Them-desgin-refactor Db-Plugin May 27, 2025
@GergesBernaba1 GergesBernaba1 mentioned this pull request May 27, 2025
@GergesBernaba1 GergesBernaba1 changed the title Db-Plugin Db2-Plugin May 27, 2025
…t table counts

- Added a new module `fixSchemaListIssue.js` to address the hanging issue in the `/database-connections/schema-list` endpoint and to refresh table counts correctly.
- Enhanced the DB2 driver in `index.js` to apply the new fixes during initialization.
- Created `schemaHelper.js` for helper functions related to schema operations, including refreshing schema counts.
- Introduced `schemaRefresh.js` to provide an improved implementation for refreshing schema counts.
- Updated the frontend driver configuration to include a global timeout for operations.
- Developed comprehensive test scripts (`test-all-db2-fixes.js`, `test-schema-fixes.js`, `test-sql-endpoint-fix.js`) to validate the fixes and ensure proper functionality.
- Added a verification script (`verify-fixes.js`) to check for required files and run tests with a mock DB2 connection.
@janproch

Copy link
Copy Markdown
Member

I am testing this plugin with local DB2 instance, there are lot of errors in console and I see no tables in table view.

How did you configure connection to DB2?

I use following settings:

image

@GergesBernaba1

Copy link
Copy Markdown
Author

I use the database URL in the DB2 connection for a database directly, and all data is retrieved successfully, and the UI shows correct
DB2

@janproch

Copy link
Copy Markdown
Member

Could you please share your database URL, of course without sensitive data?

We have no experience with DB2 (I and nobody of my colleagues), so we are struggling with this a bit

Ideally, also a screenshot of connection settings with erased sensitive data

Thank yout

@GergesBernaba1

Copy link
Copy Markdown
Author

Please check this screenshot for the connection string

image

@GergesBernaba1

Copy link
Copy Markdown
Author

this after add new rows
image

@janproch

Copy link
Copy Markdown
Member

Great thanks, I am connected now!

Sometimes it shows correct structure, sometimes not...

Probably problem is in method incrementalAnalysis in Analyser ? If this method will be deleted, it will behave better

We have to do more test, but basically it seems to work

As your plugin has GPL license, I have to ask you, whether it is OK for you to be shipped with commercial DbGate Premium?
Of course, it will be also part of opensource Community

We have lot of tests for each DB engine (approx. 100), I could merge DB2 plugin into DbGate, but with - "Experimental" status, until all these tests will be implmeneted - passed or marked as skip

Thank you

@janproch

Copy link
Copy Markdown
Member

Also, when you choose "All schemas", the are 8 of procedures without name

image

This must be returned from Analyser class

@GergesBernaba1

Copy link
Copy Markdown
Author

Thanks for the update! I'll investigate the incremental incrementalAnalysis of the issue and the bug where eight procedures lack names under "All schemas." I'm happy to help with testing to stabilize the plugin.

I'm fine with the GPL-licensed plugin being included in both DbGate Community and Premium editions. Please let me know if any licensing details need discussion.

Labeling the plugin "experimental" until tests are complete makes sense. I can assist with test implementation or debugging. Let me know the next steps!

@GergesBernaba1

Copy link
Copy Markdown
Author

Hi Jan,

I've addressed the issue in a new pull request (#1135).
Could you please review it when you have a moment?
If everything looks good, I'd recommend merging this new PR instead of the previous one.

image

@GergesBernaba1

GergesBernaba1 commented Jun 6, 2025

Copy link
Copy Markdown
Author

Hi @janproch
i hope you are doing well!
Did you check it?

@janproch

janproch commented Jun 6, 2025

Copy link
Copy Markdown
Member

Hi @GergesBernaba1 ,

yes I did, thank you
I am now a bit stucked with new release, which is planned to June 16, but I want to merge your new PR into this release, so that DB2 support will be part of this (also with upcoming Firebird support)
I believe that we will have capacity to merge and test this next week

Thank you

@GergesBernaba1

Copy link
Copy Markdown
Author

I’d be glad to contribute more to this project. Do you have a task list I can help with? Feel free to assign me any items.

@janproch

janproch commented Jun 9, 2025

Copy link
Copy Markdown
Member

Thanks for the offer. There is lot of unsolved issue for DbGate - https://github.com/dbgate/dbgate/issues

If you choose something, better to let me know, some issues are already in progress

Thanks

BTW. I have merged your newer DB2 PR into new feature branch - feature/db2, I will run some tests in this branch and then merge it into master - PR https://github.com/dbgate/dbgate/issues

@janproch

janproch commented Jun 9, 2025

Copy link
Copy Markdown
Member

No good news, we have three times red https://github.com/dbgate/dbgate/actions/runs/15536757630 ...
I tried to update electron (https://www.npmjs.com/package/ibm_db claims that it supports electron 32-35), but yet without success

@GergesBernaba1

Copy link
Copy Markdown
Author

@janproch, Are you trying to update the Electron version to: 32.1.0!

@GergesBernaba1

Copy link
Copy Markdown
Author

Could run the action again!

@janproch

Copy link
Copy Markdown
Member

I tried change version - 32.1.0, the same error - https://github.com/dbgate/dbgate/actions/runs/15577442183/job/43864975613

Build process is building native img_db package, but it should use precompiled version, probably precompiled version is not found

Probably right electron version with precompiled packages should solve this. But unfortunately, I have no capacity to solve this before current release, so probably we have to postpone adding DB2 to next release

@GergesBernaba1

Copy link
Copy Markdown
Author

image
It works fine, check the screenshot.

@GergesBernaba1

Copy link
Copy Markdown
Author

Hi @janproch
Are you trying it again? In my local machine, on Windows, the app is working without any errors

@janproch

Copy link
Copy Markdown
Member

On my PC it also worked correctly in local development.

But we need to correctly build it on github pipelines, this is a bit more complicated, because imb_db package uses native JS module, which brings problems always. This job could not be easily delegated any of my colleagues, so it must wait, until there are no urgencies from Premium users and from DbGate investor

@janproch

Copy link
Copy Markdown
Member

Hi @GergesBernaba1 ,

thanks for patience

There is apparantly problem with installing this package in electron, docker build works correctly - https://github.com/dbgate/dbgate/actions/runs/16340367906 , but electron build fails when installing packages - https://github.com/dbgate/dbgate/actions/runs/16340367905

I replicated this problem on my local computer with Windows OS, using commands:

node adjustPackageJson.js
cd app
yarn install

According to documentation, there should be precompiled packages, for electron, but they are not used

Electron Binaries - From electron v32 to v35. Hence, no support for vscode older than v1.95.

@janproch

Copy link
Copy Markdown
Member

@GergesBernaba1 It worked locally for you even under electron ?

Did you install packages for electron? ( yarn install in "app" subdirectory)

@GergesBernaba1

Copy link
Copy Markdown
Author

yes i did

@janproch

janproch commented Jul 18, 2025

Copy link
Copy Markdown
Member

If it works for you, could you please send me console log, when adding ibm_db package to electron app?

cd app
yarn add ibm_db

I ended with following error, I would like to know, whether your install process used precompiled package for electron:

yarn add v1.22.19
[1/4] Resolving packages...
warning ibm_db > q@1.5.1: You or someone you depend on is using Q, the JavaScript Promise library that gave JavaScript developers strong feelings about promises. They can almost certainly migrate to the native JavaScript promise now. Thank you literally everyone for joining me in this bet against the odds. Be excellent to each other.

(For a CapTP with native promises, see @endo/eventual-send and @endo/captp)
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "electron-builder > app-builder-lib@25.1.8" has unmet peer dependency "electron-builder-squirrel-windows@25.1.8".
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 35 new dependencies.
info Direct dependencies
└─ ibm_db@3.3.2
info All dependencies
├─ adm-zip@0.5.16
├─ available-typed-arrays@1.0.7
├─ axios@1.10.0
├─ big-integer@1.6.52
├─ bindings@1.5.0
├─ bl@1.2.3
├─ buffer-alloc-unsafe@1.1.0
├─ buffer-alloc@1.2.0
├─ buffer-fill@1.0.0
├─ call-bind-apply-helpers@1.0.2
├─ call-bind@1.0.8
├─ call-bound@1.0.4
├─ dunder-proto@1.0.1
├─ es-object-atoms@1.1.1
├─ file-uri-to-path@1.0.0
├─ follow-redirects@1.15.9
├─ for-each@0.3.5
├─ fs-constants@1.0.0
├─ has-tostringtag@1.0.2
├─ ibm_db@3.3.2
├─ is-callable@1.2.7
├─ is-typed-array@1.1.15
├─ math-intrinsics@1.1.0
├─ nan@2.23.0
├─ possible-typed-array-names@1.1.0
├─ proxy-from-env@1.1.0
├─ pump@1.0.3
├─ q@1.5.1
├─ set-function-length@1.2.2
├─ tar-fs@1.16.5
├─ tar-stream@1.6.2
├─ targz@1.0.1
├─ to-buffer@1.2.1
├─ typed-array-buffer@1.0.3
└─ which-typed-array@1.1.19
$ yarn rebuild && patch-package
yarn run v1.22.19
$ electron-builder install-app-deps
  • electron-builder  version=25.1.8
  • loaded configuration  file=package.json ("build" field)
  • executing @electron/rebuild  electronVersion=30.0.2 arch=x64 buildFromSource=false appDir=./
  • installing native dependencies  arch=x64
  • preparing       moduleName=ibm_db arch=x64
  ⨯ (node:31536) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Error: Could not find any Visual Studio installation to use
    at VisualStudioFinder.fail (C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:122:47)
    at C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:75:16
    at VisualStudioFinder.findVisualStudio2013 (C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:380:14)
    at C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:71:14
    at VisualStudioFinder.findVisualStudio2015 (C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:364:14)
    at C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:67:12
    at failPowershell (C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:156:7)
    at VisualStudioFinder.parseData (C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:170:14)
    at C:\jenasoft\dbgate-merged\app\node_modules\node-gyp\lib\find-visualstudio.js:143:14
    at ChildProcess.exithandler (node:child_process:407:7)
    at ChildProcess.emit (node:events:518:28)
    at maybeClose (node:internal/child_process:1101:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)


  ⨯ node-gyp failed to rebuild 'C:\jenasoft\dbgate-merged\app\node_modules\ibm_db'  failedTask=installAppDeps stackTrace=Error: node-gyp failed to rebuild 'C:\jenasoft\dbgate-merged\app\node_modules\ibm_db'
    at ChildProcess.<anonymous> (C:\jenasoft\dbgate-merged\app\node_modules\@electron\rebuild\lib\module-type\node-gyp\node-gyp.js:121:24)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

@GergesBernaba1

Copy link
Copy Markdown
Author

yarn add ibm_db
yarn add v1.22.22
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "electron-builder > app-builder-lib@25.1.8" has unmet peer dependency "electron-builder-squirrel-windows@25.1.8".
[4/4] Building fresh packages...
success Saved 1 new dependency.
info Direct dependencies
└─ ibm_db@3.3.2
info All dependencies
└─ ibm_db@3.3.2
$ yarn rebuild && patch-package
yarn run v1.22.22
$ electron-builder install-app-deps
• electron-builder version=25.1.8
• loaded configuration file=package.json ("build" field)
• executing @electron/rebuild electronVersion=22.3.27 arch=x64 buildFromSource=false appDir=./
• installing native dependencies arch=x64
• preparing moduleName=ibm_db arch=x64
• finished moduleName=ibm_db arch=x64
• preparing moduleName=odbc arch=x64
• finished moduleName=odbc arch=x64
• completed installing native dependencies
Done in 7.20s.
patch-package 6.5.1
Applying patches...
No patch files found
Done in 91.04s.

@GergesBernaba1

Copy link
Copy Markdown
Author

i have to set this before run it:
$env:IBM_DB_HOME="{current Loction for app}\dbgate\app\node_modules\ibm_db\installer\clidriver"

and add those if not in the package JSON file:
"ibm_db": "^3.3.2",
"odbc": "^2.4.9",

And in the main, in the same file:
"@electron/rebuild": "^3.2.10",
"node-gyp": "^11.2.0"
"electron": "22.3.27",
and downgrade the electron version, or pull the latest

@GergesBernaba1

Copy link
Copy Markdown
Author

@janproch are you checking it?

@janproch

Copy link
Copy Markdown
Member

I am trying to upgrade to newest stable electron 37 now, I hope it will help solve issues like this

@GergesBernaba1

Copy link
Copy Markdown
Author

@janproch Are you still waiting for any modifications from me?

@janproch

janproch commented Nov 3, 2025

Copy link
Copy Markdown
Member

@GergesBernaba1 plugin is OK, the problem is with the DB2 native NPM package, which breaks electron app installation. I have no idea how to solve this, probably big upgrade of all dependencies could help, but for this big upgrade we have curreclty no free capacity

@GergesBernaba1

GergesBernaba1 commented Nov 3, 2025

Copy link
Copy Markdown
Author
image but it works with me on my local machine

@janproch

janproch commented Nov 4, 2025

Copy link
Copy Markdown
Member

Problem is the build process

image

But the number of probles regarding old electron is growing (eg. #1243 ), maybe we have to make finally un upgrade, which probably will solve problem with DB2 plugin build

@janproch

janproch commented Nov 11, 2025

Copy link
Copy Markdown
Member

Electron was succesfully upgraded, but ibm_db driver doesn't seem to install. I also upgraded to newest version ibm_db, without success. Installing this package fails on build server:

image

https://github.com/dbgate/dbgate/actions/runs/19261459987/job/55067011732

@GergesBernaba1

GergesBernaba1 commented Dec 26, 2025

Copy link
Copy Markdown
Author

@janproch

Please review this PR: #1309
I've resolved the workflow issues.

@janproch

janproch commented Jan 5, 2026

Copy link
Copy Markdown
Member

@GergesBernaba1 thanks for the update

I added GH action
- name: Install ODBC
run: sudo apt-get update && sudo apt-get install -y unixodbc-dev

But there is the same problem when running yarn install

https://github.com/dbgate/dbgate/actions/runs/20719819077/job/59480310086

image

@GergesBernaba1

Copy link
Copy Markdown
Author

@janproch could you run it again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants