Tags: gorpipe/gor
Tags
fix(ENGKNOW-3691): accept db credentials from the host application (#134 ) * fix(ENGKNOW-3691): parse rda db source from APPSERVER_RDA_* environment variables * fix(ENGKNOW-3691): install env db sources before file sources, tolerate missing file * fix(ENGKNOW-3691): narrow rda env-source helpers to package-private The design spec calls for no public signature changes beyond the new package-private overload of initializeDbSources. parseEnvForDbSourceInstallation and driverClassForUrl were left public despite having no callers outside org.gorpipe.gor.model (only DbConnectionCache itself and UTestDbSource, both in-package, use them). Drop the public modifier on both, keeping them static; parseLinesForDbSourceInstallation is untouched since it is genuine public API used elsewhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3691): bump version to 5.11.2 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3691): load user connections from gor.db.credentials gor.sql.credentials is retired: gor.db.credentials now carries the additional db resources it was intended for. resolveSqlCredPath falls back to the db credentials path when gor.sql.credentials is unset, so userConnections keeps loading without it. Without this, dropping the property would silently empty userConnections on gor-worker and break every sql:// source. An explicitly configured gor.sql.credentials still wins, so existing setups are unaffected. Also documents which cache each credential source feeds: rotating credentials arrive through APPSERVER_RDA_* for systemConnections, while the credentials file supplies the additional databases reached through userConnections. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3691): remove gor.sql.credentials The db credentials file now carries the additional databases gor can reach, which is exactly what gor.sql.credentials existed for, so the separate source is redundant. Both initInConsoleApp and initInServer now load systemConnections and userConnections from gor.db.credentials. Removes the property, the ~/gor.sql.credentials home-file lookup, and the resolveSqlCredPath fallback introduced earlier on this branch. This is a breaking change for anyone setting the property. Migrates the three tests that configured it (UTestSQLInputSource, UTestInputSourceParsing, UTestSqlSource) to gor.db.credentials, and updates the DbConnection class docs, the database design note, and the SQL/GORSQL/NORSQL command docs. Also treats a blank APPSERVER_RDA_PASSWORD as unset, matching how url and username are already handled. A secret manager or template rendering an empty value would otherwise install it as a real empty password. Covered at both the parse and install level. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3691): rename env credential vars to GREGOR_DB_* APPSERVER_RDA_URL/USERNAME/PASSWORD/DRIVER become GREGOR_DB_URL/USERNAME/ PASSWORD/DRIVER. The old names were borrowed from SM, which GOR does not otherwise depend on; the GREGOR_ prefix keeps them distinct from generic DB_* vars that a base image, sidecar, or platform default might inject. The installed source is still named "rda". LORD_DB_* is unrelated and untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3691): take db credentials from the caller, not the environment Gor no longer reads credentials from the environment. Instead the host application passes them in as DbCredentials, a new public record, via DbConnectionCache.initializeDbSources(String, List) or DbConnection.initInServer(List). Supplied credentials are installed before the file is read, so a file row of the same name still takes precedence. This keeps deployment-specific env var naming out of the library: gor does not care whether the host sourced them from its own config, a secret manager, or environment variables it names itself. toPartsForInstallation converts supplied credentials into the same shape the file parser produces, so installDbSourceFromParts remains the single place that loads drivers and constructs DbConnection. Driver derivation from the url prefix, blank-as-unset handling, and skip-with-warning on incomplete input are unchanged in behaviour, just moved off the env path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ENGKNOW-3691): rewrite the database configuration section The Configuration section was a block of javadoc pasted into markdown - li tags, pre blocks, and a TBD - and still described two credentials files. Rewrites it as markdown covering the two sources that exist now: the gor.db.credentials file and credentials the host passes in as DbCredentials, with the precedence rule between them and the missing-file behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3691): Update docs. * fix(ENGKNOW-3691): feed the system and user caches from separate sources systemConnections now comes from the credentials the host passes in, and userConnections from gor.db.credentials. Previously both caches were fed from both sources with the file taking precedence. Keeping them apart matches what each is for: system credentials rotate and cannot be baked into a file without going stale, and the rotating system credentials are not the ones user queries should reach. It also removes the collision between the two, so there is no precedence rule left to get wrong. The two initializeDbSources overloads are now alternatives rather than a merge - initializeDbSources(String) reads the file, initializeDbSources(List) installs supplied credentials, and each clears the cache first. initInServer's parameter is renamed systemCredentials to say which cache it feeds. A console app has no host to supply credentials, so initInConsoleApp still loads both caches from the file. Note passing no credentials now leaves the system cache empty rather than falling back to the file, so db:// will not resolve in a server that supplies none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(ENGKNOW-3691): say which connection cache each access method uses Now that the system and user caches are fed from different sources, which cache an access method resolves against determines which credentials it gets - so the document has to say. Adds that to the notes for each of the four methods: sql commands and sql:// resolve against the user connections, db:// and //db: against the system connections. The SQL, GORSQL and NORSQL command pages now say the databases they reach are the user connections rather than just "the database". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(ENGKNOW-3657): harden DictionaryEntries read accessors against cl… …ear() races (#130) * docs(ENGKNOW-3657): add table-service memory diagnosis design spec Diagnosis-first spec: local load harness + profiler to pinpoint the dominant memory consumer behind table-service OOMs under load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(ENGKNOW-3657): add table-service memory diagnosis implementation plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): add MemoryLoadConfig knobs for memory harness Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): add MemorySampler for peak heap/RSS tracking Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): make MemorySampler peak tracking race-free Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): add DictionaryFixture to build dictionaries at scale Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): add TableServiceLoadDriver for mixed read+write load Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): write through locked table transaction in load driver Use the same exclusive-lock write path as the real Table Service so concurrent writes serialize instead of racing on a shared temp file and silently losing inserts. Log caught op errors; shutdownNow on await timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): add TableServiceLoadMain profiler entrypoint Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): add missing copyright header to load-main test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): validate harness stresses dictionary memory path Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): model dictionary retention and measure retained heap Cache table instances so read-side DictionaryEntries stay retained for the run (models a session/table cache), and measure post-GC retained-heap delta over a baseline. Replaces the peak-churn metric that failed to scale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(ENGKNOW-3657): correct shared-read concurrency comments in load driver Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ENGKNOW-3657): add S3-backed dictionary fixture variant Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(ENGKNOW-3657): add profiling runbook and findings template Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ENGKNOW-3657): make dictionary TableCache tunable and record stats Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): build dictionary content map lazily contentHashToLines, activeTags and deletedEntriesCount are only needed by the mutation/stats paths, not by select/filter reads. Building them lazily shrinks the long-lived cached dictionary copy for select-heavy workloads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): Remove redundant assert. * fix(ENGKNOW-3657): Tune max cache size. * chore(ENGKNOW-3657): remove planning docs from VCS Keep the memory-diagnosis design spec and implementation plan as local-only files; not part of the PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ENGKNOW-3657): remove findings and runbook from VCS Keep the memory findings and profiling runbook as local-only files; not part of the PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(ENGKNOW-3657): add profiling runbook Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): safe publication for concurrent dictionary reads getEntries()/isLoaded() read dataLoaded+rawLines without synchronization while a synchronized loader publishes them, so a concurrent reader could see dataLoaded==true with a stale rawLines (transient NPE/torn read). This is hit by concurrent selects sharing one cached GorDictionaryTable. Mark the lazily-published DictionaryEntries fields (and TableInfoBase.id) volatile so unsynchronized reads observe fully-constructed state; correct the class contract in the javadoc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): safe lazy init of table access optimizer getTableAccessOptimizer() built the optimizer via an unsynchronized null-check on a non-volatile field, so concurrent callers on a shared table could build/publish multiple instances unsafely. Use double-checked locking on a volatile field so concurrent readers share one safely-published optimizer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): safe publication for lazy contentType cache inferShouldBucketizeFromContent() lazily cached the content type in a non-volatile field with an unsynchronized null-check. Make the field volatile and compute via a local; the computation is idempotent so no lock is needed, and reads now observe a safely-published value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ENGKNOW-3657): Fix possible threading use with clear/getEntries. * test(ENGKNOW-3657): regression tests for dictionary read-accessor hardening Cover the getEntries()/getAllActiveTags() unmodifiable-view guarantees, the updateTagMap() reload-after-clear guard, and that insert/delete still mutate the backing list after switching to the internal loadedEntries() accessor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(ENGKNOW-3363): fix security vuln in exec gor git (#119) * fix(ENGKNOW-3363): Add access check methods based on strings (in addition to the ones based on AuthorizationAction enum). * fix(ENGKNOW-3363): Test bug fixes in the exec commands * fix(ENGKNOW-3363): Fix/improve file commands. * fix(ENGKNOW-3363): Improving Git wrapper. * fix(ENGKNOW-3363): Improving Git wrapper. * fix(ENGKNOW-3363): Improving Git wrapper. * feat(ENGKNOW-3363): Add link updateMeta command. * feat(ENGKNOW-3363): Add link updateMeta command. * fix(ENGKNOW-3363): Hiding rmCommand * fix(ENGKNOW-3363): Hiding rmCommand * fix(ENGKNOW-3363): Fix security vuln.
fix(ENGKNOW-3046): enable cram binding to correct reference genome fo… …r exome cloud bfx (#103) * feat(ENGKNOW-3046): Enable multiple cram reference files. * feat(ENGKNOW-3046): Enable multiple cram reference files. * feat(ENGKNOW-3046): Enable multiple cram reference files. * feat(ENGKNOW-3046): Enable multiple cram reference files. * feat(ENGKNOW-3046): Enable multiple cram reference files. * feat(ENGKNOW-3046): Bump version * feat(ENGKNOW-3046): Enable multiple cram reference files.
PreviousNext