Repository navigation
Branch 3.4.0 - #1371
Open
paynie wants to merge 17 commits into
Open
Branch 3.4.0#1371paynie wants to merge 17 commits into
paynie wants to merge 17 commits into
Conversation
Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1347]Feat: new features of Ego Network algorithm
…h algo docs Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1361]Docs: fix invalid references and bump version to 3.4.0 in graph algo docs
…4 dependency Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1364]Feat: new Leiden community detection algorithm and remove algs4 dependency
… in READMEs Signed-off-by: xiaolongwen <xiaolongwen@tencent.com>
[#1366]Docs: reorganize Angel Graph section and enrich algorithm list in READMEs
…dd standalone PS deployment and stability fixes Overview - Move the SONA graph algorithm layer from spark-on-angel-graph to spark-on-angel-mllib (package com.tencent.angel.graph.* -> com.tencent.angel.spark.ml.graph.algo.*); spark-on-angel-mllib now depends on spark-on-angel-graph instead of the reverse. - Verified end to end on Hadoop 2.10.2 + Spark 3.3.4 (yarn-cluster) with 45 graph / embedding / ANN examples. Features - Graph algorithms: SCC, BatchedCC, CompressCC, CutCC, LPAX, LouvainCPM / Density / Repair / Opt, BGE, LINE v2/v3, MetaPath2Vec V1/V2, DocVec, EmbeddingPooling, ClusterRank, KCoreDC, KCoreCompress, PPR, WHIndex, Intimacy, click-rate features (OneHop, OneHopCom, OneHopWeighted, TwoHopCom, LBSBased), edge features, LinkPrediction, PostIndex, KCliqueV2, directed triangle / LCC, SSSP / MSSP, IVFHnsw, KMeans-IVFHnsw. - ML modules: time series (ARIMA, GARCH, Holt-Winters, ...), KD-Tree, IsolationForest / LOF / KNN, Apriori / FPGrowth / PrefixSpan, feature preprocessing, word segmentation, PMML export, ALS. - PS: STANDALONE / EMBEDDED deployment modes and attach mode with launch scripts under angel-ps/bin; metrics framework; FlatBuffers serialization; new graph storage, sampling and label/timestamp PSFs. - Transport: heartbeats, ordered retry queue, request deadline, bounded thread pools, idle connection reclamation, PSF timeout (-Dangel.psf.timeout.ms). - Examples: 70 cluster and 74 local examples, ClusterExampleUtils, MixedLR / NFM / PNN json configs, and test data under data/. - Add RELEASE_NOTES_3.4.0.md and RELEASE_NOTES_3.4.0_CN.md. Bug fixes - YarnContainerAllocator: custom ResourceRequest comparator to avoid NPE on Hadoop 2.9+/2.10. - Transport: fix resource leak on fatal/abandoned requests, response vs timeout race, no-op Thread.stop(), dispatcher dying on one exception, throttled recovery events, ChannelManager2 checker never started, port bind races. - Storage: fix deepClone returning empty maps, dataSize NPE under concurrent rehash, silently loading truncated snapshots. - Graph: wait on psfUpdate futures (LinkPrediction, SLPA, COPRA, KCoreDC, LINE, LINE v3); use RangePartitioner for OneHopWeighted and ClusterRank matrices; fix empty LINE v3 text output and OneHopWeighted ClassCastException. - Deploy: replace netty-all-4.1.42 / stream-2.7.0 in spark-on-angel-env.sh with jars shipped in the release; pin jackson 2.13.4 for Spark 3.3; LINEExample only sets master when mode is given. Signed-off-by: paynie <cswjsxs2005@gmail.com>
…n 2 Spark tasks and PS partitions Verified on YARN cluster mode (2 PS, partitionNum=4, psPartitionNum=4): all 46 algorithms succeed, every non-trivial Spark stage runs > 2 tasks and every PS matrix has > 2 partitions. PS partitioning - RangePartitioner: clamp zero block row/col size to 1 to avoid an infinite loop (master hang and RPC timeout) when partitions outnumber rows/cols - ModelContextUtils / EmbeddingPoolingModel: use ceil division with a minimum of 1 for block size - SubSampling/LINEExample: honor psPartitionNum for the freq matrix Spark parallelism - Pass minPartitions to textFile/GraphIO.loadString when reading input (ClusterExampleUtils, BGE, LINE, LINEV2, LINEV3, Word2vec, DocVec, EmbeddingPooling, EGESRemapping, NeighborTabJoinSideInfoTab, ann DataUtils) - NEModel: sliced saving uses max(executors, spark.default.parallelism) partitions - RePartition: write partition bounds with partitionNum tasks instead of 1 Correctness with more partitions - FunctionalEdgeAggregate: compute neighbor degrees globally and join them to edges; looking them up within one partition caused ArrayIndexOutOfBounds (-1) - NeighborTablePartition.testPS: check min(num, size) nodes instead of asserting that every partition has at least num nodes - ANNPSModel / IVFHnswPSModel: skip PS calls for empty query partitions, which produced PSF requests with no splits Signed-off-by: paynie <cswjsxs2005@gmail.com>
Verified in Spark local mode: 88 of 89 examples under examples/local succeed
(AngelClientTest needs an external PS address and is out of scope). The YARN
cluster run (46 algorithms, partitionNum=4, psPartitionNum=4) still passes
with these changes.
Library fixes
- DataParser: extend Serializable so its case-class subclasses can be
deserialized by Spark ("no valid constructor" in OnlineExample)
- OnlineLearner: skip empty micro-batches and empty partitions instead of
issuing PS pulls with null column indices
- HyperANFPSModel: skip PSF calls for empty node arrays/updates, which
produced PSF requests with zero splits (HyperDecimal)
- MultiAttrHIndexPSModel.initAttributes: use fold(0L) so an empty RDD returns
0 instead of throwing "empty collection"
Example fixes
- GBDTExample (local): set storageLevel, which was left null
- SSHopLayoutLocalExample / WHIndexLocalExample: default
spark.executor.extraJavaOptions to "" when it is not set
- BiGraphAggre / MultiAttrAggregate / MultiAttrHIndex / TopCandidate local
examples: use the range PS router, since node attribute PSFs
(GetNodeAttrs) only support range partitioning
Signed-off-by: paynie <cswjsxs2005@gmail.com>
…loyment modes Replace legacy yarn-cluster masters with yarn across SONA examples and align mode documentation. Update SONA-example to use explicit cluster deploy mode and executor JVM options. Document local, YARN, STANDALONE, EMBEDDED and Attach deployment in Chinese and English, including dependency conflicts, readiness and lifecycle constraints. Signed-off-by: paynie <cswjsxs2005@gmail.com>
… tests Implement matrix stream routing, enforce range partitioning for CSR neighbors, and support CSR snapshot save/load with checkpoint recovery regressions. Propagate local master startup failures, bound startup waits, and clean up failed local clusters. Exclude conflicting Hadoop 3 client dependencies in local-test and run ScalaTest in a dedicated JVM. Make test fixtures platform-independent, handle optional LINE embedding paths safely, and restore Word2Vec initialization and loss verification. Validation: all 14 modules build successfully; PS core 421 passed with 1 SSH-dependent test skipped, graph regressions 3 passed, SONA core 25 passed, and SONA mllib 18 passed. Signed-off-by: paynie <cswjsxs2005@gmail.com>
…gorithms Reconnect Kubernetes submission, shared Master lifecycle, PS/Worker attempt scheduling, retries and resource cleanup while preserving other deployment modes. Package Kubernetes image scripts, normalize API URL ports, avoid Hadoop 2/3 classpath conflicts, and isolate native Fabric8 dependencies from Spark using angel.kubernetes.client.classpath. Resolve late Worker task configuration, pull prediction parameters for initial and tail batches, and propagate GraphRunner prediction failures. Add Kubernetes factory, pod specification and submission bridge regressions; publish the classified Angel parameter handbook and reference with deployment guidance and validation boundaries. Validation: all 14 modules build; 16 targeted unit tests pass. Single-node Kubernetes 1.31 E2E covers matrix save/load and PS replacement, native LR train/save/load/predict, and PageRank on Spark 3.3.4 driver/executor pods via attach and direct PS launch. Algorithm outputs checked against independent references; multi-node and all-algorithm coverage remain unvalidated. Signed-off-by: paynie <cswjsxs2005@gmail.com>
…trips Align Json4s with Spark 3.3 and initialize optional PMML converters only when export is requested; retain explicit PMML export compatibility limits. Preserve unknown-dimension sparse storage, save columns using actual row storage offsets for hash/range routing, and prevent the model saver dispatcher from busy-spinning. Parse PSF responses once, complete low-level futures correctly, and retain bounded CSR payload slices with correct pooled-buffer cursor and ownership handling to unblock LDA. Handle empty neighbor requests, remove time-series UDF closure capture, correct output schemas and graph example context/partition handling, and explicitly stop JSON/offline runner resources. Parameterize basic PS examples and implement real logistic gradients with LIBSVM index bounds, valid Adam update steps and numerical assertions; add focused regression tests. Validation: 14-module build and final example rebuild pass; core Java 421 passed/1 skipped, core Scala 14 passed, SONA core 25 passed, SONA mllib 20 passed, graph 3 passed. Original failing Local/YARN cases and KMeans/LDA workflows pass. Independent checks verify model prediction row counts, nonempty mutual-friend recall and Basic LR logloss reduction from 0.693147 to 0.562404. Corrupted old KMeans checkpoints require regeneration; SSH/external PS prerequisites remain unchanged. Signed-off-by: paynie <cswjsxs2005@gmail.com>
…ass loading Validate and normalize client supplied model paths at the Master save/load RPC boundary: reject empty paths, parent traversal and unsupported file authorities, and optionally confine paths to configured roots. Prevent model saves from recursively deleting arbitrary existing directories: the final commit only replaces genuine Angel model directories (or an explicitly enabled overwrite), writes a job ownership marker, and verifies it before loading. Legacy model directories without a marker remain readable with a warning. Restrict dynamic class loading: RPC protocol names must resolve to VersionedProtocol interfaces and model format classes to concrete MatrixFormat implementations inside the Angel packages, with configuration based allow lists for extensions. Bind each PS attempt to its first registered location so heartbeat re-registration stays idempotent but conflicting addresses are rejected; add a state-machine guard and an optional caller source address check. Pin Jackson core/databind/annotations to the Spark 3.3 runtime versions to remove the test classpath conflict introduced by Json4s 3.7.0-M11, and add ModelSecurityTest covering traversal, safe overwrite, ownership markers, class allow lists and registration conflicts. Validation: angel-ps-core compiles and jars; 12 new security tests pass; core Java 386 passed/9 skipped with 0 failures and 0 errors, core Scala 14 passed; Local Spark ML train+predict and JsonRunner train/save, predict/load and re-save all pass with the ownership marker written; YARN JsonRunner train succeeds; git diff --check is clean. Master RPC still has no cryptographic authentication; strict allowed roots, source address checking and full credential support remain deployment/protocol hardening work. Signed-off-by: paynie <cswjsxs2005@gmail.com>
Drop 23 .fbs schema files from angel-ps/core/src/main/fbs that are not part of the current build or runtime path: - 15 optimizer-specific schemas (Adadelta, Adagrad, Adam, FTRL, Lion, Momentum, Nadam, RMSprop, SGD, etc.) - shared gradient, training, quantization, hyper-parameter, init, common and frequency-stat schemas that were either referenced only from the removed set or had no references at all Keep weight_message.fbs and stats_param.fbs. They are the source of the six checked-in FlatBuffers generated classes under com.tencent.angel.flatbuffers.dollar.tensor, and WeightMessage is used by AdsEmbeddingOutputFormat for embedding model output. stats_param.fbs is still included by weight_message.fbs and backs TaskStats. Validation: - no Maven plugin or script runs flatc over src/main/fbs; angel-ps/core/pom.xml only declares the flatbuffers-java 23.5.26 runtime for the checked-in stubs - git grep finds no .fbs references outside src/main/fbs and no Java/Scala references to the removed message types - all six generated Java stubs remain; the remaining include graph is weight_message.fbs -> stats_param.fbs with no dangling includes - git status shows exactly 23 deletions and no other tracked changes .fbs files are not compiled by Maven, so no code-generation or test rerun was required for this cleanup. Untracked validation artifacts were excluded from the commit. Signed-off-by: paynie <cswjsxs2005@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist:
Please check each of these after submitting your pull request:
[Issue ID]