Skip to content

Branch 3.4.0 - #1371

Open
paynie wants to merge 17 commits into
masterfrom
branch-3.4.0
Open

paynie wants to merge 17 commits into
masterfrom
branch-3.4.0

Conversation

@paynie

@paynie paynie commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Pre-submission checklist:

Please check each of these after submitting your pull request:

  • Do you submit an issue before submitting your pull request, each pull request is supposed to connect to an issue. If your do not submit an issue, please submit one at first.
  • Do you connect you pull request to the issue. The title of pull request should start with [Issue ID]
  • Do you assign a code reviewer? If your don't know how to select code reviewers, please refer CODEOWNERS
  • If you pull request is big, it would be hard to merge by the community at once, please contact the TSC member in advance.

xiaolongwen and others added 17 commits September 11, 2026 11:10
Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1347]Feat: new features of Ego Network algorithm
…h algo docs

Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1361]Docs: fix invalid references and bump version to 3.4.0 in graph algo docs
…4 dependency

Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1364]Feat: new Leiden community detection algorithm and remove algs4 dependency
… in READMEs

Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1366]Docs: reorganize Angel Graph section and enrich algorithm list in READMEs
…dd standalone PS deployment and stability fixes

Overview
- Move the SONA graph algorithm layer from spark-on-angel-graph to
  spark-on-angel-mllib (package com.tencent.angel.graph.* ->
  com.tencent.angel.spark.ml.graph.algo.*); spark-on-angel-mllib now depends
  on spark-on-angel-graph instead of the reverse.
- Verified end to end on Hadoop 2.10.2 + Spark 3.3.4 (yarn-cluster) with 45
  graph / embedding / ANN examples.

Features
- Graph algorithms: SCC, BatchedCC, CompressCC, CutCC, LPAX, LouvainCPM /
  Density / Repair / Opt, BGE, LINE v2/v3, MetaPath2Vec V1/V2, DocVec,
  EmbeddingPooling, ClusterRank, KCoreDC, KCoreCompress, PPR, WHIndex,
  Intimacy, click-rate features (OneHop, OneHopCom, OneHopWeighted,
  TwoHopCom, LBSBased), edge features, LinkPrediction, PostIndex,
  KCliqueV2, directed triangle / LCC, SSSP / MSSP, IVFHnsw, KMeans-IVFHnsw.
- ML modules: time series (ARIMA, GARCH, Holt-Winters, ...), KD-Tree,
  IsolationForest / LOF / KNN, Apriori / FPGrowth / PrefixSpan, feature
  preprocessing, word segmentation, PMML export, ALS.
- PS: STANDALONE / EMBEDDED deployment modes and attach mode with launch
  scripts under angel-ps/bin; metrics framework; FlatBuffers serialization;
  new graph storage, sampling and label/timestamp PSFs.
- Transport: heartbeats, ordered retry queue, request deadline, bounded
  thread pools, idle connection reclamation, PSF timeout
  (-Dangel.psf.timeout.ms).
- Examples: 70 cluster and 74 local examples, ClusterExampleUtils, MixedLR /
  NFM / PNN json configs, and test data under data/.
- Add RELEASE_NOTES_3.4.0.md and RELEASE_NOTES_3.4.0_CN.md.

Bug fixes
- YarnContainerAllocator: custom ResourceRequest comparator to avoid NPE on
  Hadoop 2.9+/2.10.
- Transport: fix resource leak on fatal/abandoned requests, response vs
  timeout race, no-op Thread.stop(), dispatcher dying on one exception,
  throttled recovery events, ChannelManager2 checker never started, port
  bind races.
- Storage: fix deepClone returning empty maps, dataSize NPE under concurrent
  rehash, silently loading truncated snapshots.
- Graph: wait on psfUpdate futures (LinkPrediction, SLPA, COPRA, KCoreDC,
  LINE, LINE v3); use RangePartitioner for OneHopWeighted and ClusterRank
  matrices; fix empty LINE v3 text output and OneHopWeighted
  ClassCastException.
- Deploy: replace netty-all-4.1.42 / stream-2.7.0 in spark-on-angel-env.sh
  with jars shipped in the release; pin jackson 2.13.4 for Spark 3.3;
  LINEExample only sets master when mode is given.

Signed-off-by: paynie <cswjsxs2005@gmail.com>
…n 2 Spark tasks and PS partitions

Verified on YARN cluster mode (2 PS, partitionNum=4, psPartitionNum=4): all 46
algorithms succeed, every non-trivial Spark stage runs > 2 tasks and every PS
matrix has > 2 partitions.

PS partitioning
- RangePartitioner: clamp zero block row/col size to 1 to avoid an infinite loop
  (master hang and RPC timeout) when partitions outnumber rows/cols
- ModelContextUtils / EmbeddingPoolingModel: use ceil division with a minimum of 1
  for block size
- SubSampling/LINEExample: honor psPartitionNum for the freq matrix

Spark parallelism
- Pass minPartitions to textFile/GraphIO.loadString when reading input
  (ClusterExampleUtils, BGE, LINE, LINEV2, LINEV3, Word2vec, DocVec,
  EmbeddingPooling, EGESRemapping, NeighborTabJoinSideInfoTab, ann DataUtils)
- NEModel: sliced saving uses max(executors, spark.default.parallelism) partitions
- RePartition: write partition bounds with partitionNum tasks instead of 1

Correctness with more partitions
- FunctionalEdgeAggregate: compute neighbor degrees globally and join them to
  edges; looking them up within one partition caused ArrayIndexOutOfBounds (-1)
- NeighborTablePartition.testPS: check min(num, size) nodes instead of asserting
  that every partition has at least num nodes
- ANNPSModel / IVFHnswPSModel: skip PS calls for empty query partitions, which
  produced PSF requests with no splits

Signed-off-by: paynie <cswjsxs2005@gmail.com>
Verified in Spark local mode: 88 of 89 examples under examples/local succeed
(AngelClientTest needs an external PS address and is out of scope). The YARN
cluster run (46 algorithms, partitionNum=4, psPartitionNum=4) still passes
with these changes.

Library fixes
- DataParser: extend Serializable so its case-class subclasses can be
  deserialized by Spark ("no valid constructor" in OnlineExample)
- OnlineLearner: skip empty micro-batches and empty partitions instead of
  issuing PS pulls with null column indices
- HyperANFPSModel: skip PSF calls for empty node arrays/updates, which
  produced PSF requests with zero splits (HyperDecimal)
- MultiAttrHIndexPSModel.initAttributes: use fold(0L) so an empty RDD returns
  0 instead of throwing "empty collection"

Example fixes
- GBDTExample (local): set storageLevel, which was left null
- SSHopLayoutLocalExample / WHIndexLocalExample: default
  spark.executor.extraJavaOptions to "" when it is not set
- BiGraphAggre / MultiAttrAggregate / MultiAttrHIndex / TopCandidate local
  examples: use the range PS router, since node attribute PSFs
  (GetNodeAttrs) only support range partitioning

Signed-off-by: paynie <cswjsxs2005@gmail.com>
…loyment modes

Replace legacy yarn-cluster masters with yarn across SONA examples and align mode documentation.

Update SONA-example to use explicit cluster deploy mode and executor JVM options.

Document local, YARN, STANDALONE, EMBEDDED and Attach deployment in Chinese and English, including dependency conflicts, readiness and lifecycle constraints.

Signed-off-by: paynie <cswjsxs2005@gmail.com>
… tests

Implement matrix stream routing, enforce range partitioning for CSR neighbors, and support CSR snapshot save/load with checkpoint recovery regressions.

Propagate local master startup failures, bound startup waits, and clean up failed local clusters.

Exclude conflicting Hadoop 3 client dependencies in local-test and run ScalaTest in a dedicated JVM.

Make test fixtures platform-independent, handle optional LINE embedding paths safely, and restore Word2Vec initialization and loss verification.

Validation: all 14 modules build successfully; PS core 421 passed with 1 SSH-dependent test skipped, graph regressions 3 passed, SONA core 25 passed, and SONA mllib 18 passed.
Signed-off-by: paynie <cswjsxs2005@gmail.com>
…gorithms

Reconnect Kubernetes submission, shared Master lifecycle, PS/Worker attempt scheduling, retries and resource cleanup while preserving other deployment modes.

Package Kubernetes image scripts, normalize API URL ports, avoid Hadoop 2/3 classpath conflicts, and isolate native Fabric8 dependencies from Spark using angel.kubernetes.client.classpath.

Resolve late Worker task configuration, pull prediction parameters for initial and tail batches, and propagate GraphRunner prediction failures.

Add Kubernetes factory, pod specification and submission bridge regressions; publish the classified Angel parameter handbook and reference with deployment guidance and validation boundaries.

Validation: all 14 modules build; 16 targeted unit tests pass. Single-node Kubernetes 1.31 E2E covers matrix save/load and PS replacement, native LR train/save/load/predict, and PageRank on Spark 3.3.4 driver/executor pods via attach and direct PS launch. Algorithm outputs checked against independent references; multi-node and all-algorithm coverage remain unvalidated.
Signed-off-by: paynie <cswjsxs2005@gmail.com>
…trips

Align Json4s with Spark 3.3 and initialize optional PMML converters only when export is requested; retain explicit PMML export compatibility limits.

Preserve unknown-dimension sparse storage, save columns using actual row storage offsets for hash/range routing, and prevent the model saver dispatcher from busy-spinning.

Parse PSF responses once, complete low-level futures correctly, and retain bounded CSR payload slices with correct pooled-buffer cursor and ownership handling to unblock LDA.

Handle empty neighbor requests, remove time-series UDF closure capture, correct output schemas and graph example context/partition handling, and explicitly stop JSON/offline runner resources.

Parameterize basic PS examples and implement real logistic gradients with LIBSVM index bounds, valid Adam update steps and numerical assertions; add focused regression tests.

Validation: 14-module build and final example rebuild pass; core Java 421 passed/1 skipped, core Scala 14 passed, SONA core 25 passed, SONA mllib 20 passed, graph 3 passed. Original failing Local/YARN cases and KMeans/LDA workflows pass. Independent checks verify model prediction row counts, nonempty mutual-friend recall and Basic LR logloss reduction from 0.693147 to 0.562404. Corrupted old KMeans checkpoints require regeneration; SSH/external PS prerequisites remain unchanged.
Signed-off-by: paynie <cswjsxs2005@gmail.com>
…ass loading

Validate and normalize client supplied model paths at the Master save/load RPC boundary: reject empty paths, parent traversal and unsupported file authorities, and optionally confine paths to configured roots.

Prevent model saves from recursively deleting arbitrary existing directories: the final commit only replaces genuine Angel model directories (or an explicitly enabled overwrite), writes a job ownership marker, and verifies it before loading. Legacy model directories without a marker remain readable with a warning.

Restrict dynamic class loading: RPC protocol names must resolve to VersionedProtocol interfaces and model format classes to concrete MatrixFormat implementations inside the Angel packages, with configuration based allow lists for extensions.

Bind each PS attempt to its first registered location so heartbeat re-registration stays idempotent but conflicting addresses are rejected; add a state-machine guard and an optional caller source address check.

Pin Jackson core/databind/annotations to the Spark 3.3 runtime versions to remove the test classpath conflict introduced by Json4s 3.7.0-M11, and add ModelSecurityTest covering traversal, safe overwrite, ownership markers, class allow lists and registration conflicts.

Validation: angel-ps-core compiles and jars; 12 new security tests pass; core Java 386 passed/9 skipped with 0 failures and 0 errors, core Scala 14 passed; Local Spark ML train+predict and JsonRunner train/save, predict/load and re-save all pass with the ownership marker written; YARN JsonRunner train succeeds; git diff --check is clean. Master RPC still has no cryptographic authentication; strict allowed roots, source address checking and full credential support remain deployment/protocol hardening work.
Signed-off-by: paynie <cswjsxs2005@gmail.com>
Drop 23 .fbs schema files from angel-ps/core/src/main/fbs that are not
part of the current build or runtime path:

- 15 optimizer-specific schemas (Adadelta, Adagrad, Adam, FTRL, Lion,
  Momentum, Nadam, RMSprop, SGD, etc.)
- shared gradient, training, quantization, hyper-parameter, init, common
  and frequency-stat schemas that were either referenced only from the
  removed set or had no references at all

Keep weight_message.fbs and stats_param.fbs. They are the source of the six
checked-in FlatBuffers generated classes under
com.tencent.angel.flatbuffers.dollar.tensor, and WeightMessage is used by
AdsEmbeddingOutputFormat for embedding model output. stats_param.fbs is
still included by weight_message.fbs and backs TaskStats.

Validation:
- no Maven plugin or script runs flatc over src/main/fbs; angel-ps/core/pom.xml
  only declares the flatbuffers-java 23.5.26 runtime for the checked-in stubs
- git grep finds no .fbs references outside src/main/fbs and no Java/Scala
  references to the removed message types
- all six generated Java stubs remain; the remaining include graph is
  weight_message.fbs -> stats_param.fbs with no dangling includes
- git status shows exactly 23 deletions and no other tracked changes

.fbs files are not compiled by Maven, so no code-generation or test rerun
was required for this cleanup. Untracked validation artifacts were excluded
from the commit.

Signed-off-by: paynie <cswjsxs2005@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants