Skip to content

[feature][plugin][mongodbreader] Support Nested Field Extraction via Dot Notation - #1360

Merged
wgzhao merged 1 commit into
masterfrom
add_nest_fields
Oct 5, 2025
Merged

[feature][plugin][mongodbreader] Support Nested Field Extraction via Dot Notation#1360
wgzhao merged 1 commit into
masterfrom
add_nest_fields

Conversation

@wgzhao

@wgzhao wgzhao commented Oct 5, 2025

Copy link
Copy Markdown
Owner

Motivation:

Currently, the MongoDBReader plugin only supports extracting top-level fields from MongoDB documents. However, many MongoDB schemas use nested objects, and users often need to extract specific values from these nested structures. Supporting dot notation (e.g., pdf.fit, pdf.seal_file.reveal) in the column configuration will greatly improve the plugin's flexibility and usability for real-world data scenarios.

Implementation:

  • Added a new method getNestedValue to MongoRowConverter that recursively resolves nested field paths using dot notation.
  • Updated the processOne method to utilize getNestedValue for all column extractions, enabling support for both flat and deeply nested fields.
  • Ensured type handling and error cases are consistent with the previous implementation.
  • Improved code comments for clarity and maintainability.

Result:

  • Users can now specify nested fields in the column configuration using dot notation.
  • The plugin will correctly extract and convert values from any depth of nested MongoDB documents.
  • Backward compatibility is maintained for existing configurations.
  • Code documentation is now fully in English for better collaboration.

Test Output:

$ bin/addax.sh job/mongoreader-with-nested-field.json 

2025-10-05 14:17:17.566 [        main] INFO  Engine               - 
  ___      _     _            
 / _ \    | |   | |           
/ /_\ \ __| | __| | __ ___  __
|  _  |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |>  < 
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version ::    (v5.0.1-SNAPSHOT)
2025-10-05 14:17:17.762 [        main] INFO  VMInfo               - VMInfo# operatingSystem class => com.sun.management.internal.OperatingSystemImpl
2025-10-05 14:17:17.772 [        main] INFO  Engine               - 
{
        "setting":{
                "speed":{
                        "channel":1,
                        "bytes":-1
                }
        },
        "content":{
                "reader":{
                        "name":"mongodbreader",
                        "parameter":{
                                "connection":{
                                        "database":"test",
                                        "address":[
                                                "127.0.0.1:27017"
                                        ],
                                        "collection":"col",
                                        "authDb":"admin"
                                },
                                "username":"a",
                                "password":"*****",
                                "column":[
                                        "order_type",
                                        "order_id",
                                        "update_at_time",
                                        "age",
                                        "nickname",
                                        "n1.c1",
                                        "n1.c2",
                                        "n1.c3.cc1",
                                        "n1.c3.cc2"
                                ],
                                "query":"{order_id: 'order_test_1'}"
                        }
                },
                "writer":{
                        "name":"streamwriter",
                        "parameter":{
                                "print":"true"
                        }
                }
        }
}

2025-10-05 14:17:17.807 [        main] INFO  JobContainer         - The jobContainer begins to process the job.
2025-10-05 14:17:17.905 [       job-0] INFO  JobContainer         - The Reader.Job [mongodbreader] perform prepare work .
2025-10-05 14:17:17.906 [       job-0] INFO  JobContainer         - The Writer.Job [streamwriter] perform prepare work .
2025-10-05 14:17:17.906 [       job-0] INFO  JobContainer         - Job set Channel-Number to 1 channel(s).
2025-10-05 14:17:18.098 [       job-0] INFO  JobContainer         - The Reader.Job [mongodbreader] is divided into [1] task(s).
2025-10-05 14:17:18.098 [       job-0] INFO  JobContainer         - The Writer.Job [streamwriter] is divided into [1] task(s).
2025-10-05 14:17:18.122 [       job-0] INFO  JobContainer         - The Scheduler launches [1] taskGroup(s).
2025-10-05 14:17:18.130 [ taskGroup-0] INFO  TaskGroupContainer   - The taskGroupId=[0] started [1] channels for [1] tasks.
2025-10-05 14:17:18.132 [ taskGroup-0] INFO  Channel              - The Channel set byte_speed_limit to -1, No bps activated.
2025-10-05 14:17:18.132 [ taskGroup-0] INFO  Channel              - The Channel set record_speed_limit to -1, No tps activated.

online  order_test_1   NULL    58     G0G    c1-of-n1                   cc1-of-c3-of-n1      

2025-10-05 14:17:21.141 [       job-0] INFO  AbstractScheduler    - The scheduler has completed all tasks.
2025-10-05 14:17:21.142 [       job-0] INFO  JobContainer         - The Writer.Job [streamwriter] perform post work.
2025-10-05 14:17:21.142 [       job-0] INFO  JobContainer         - The Reader.Job [mongodbreader] perform post work.
2025-10-05 14:17:21.146 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 1 records, 202 bytes | Speed 67B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2025-10-05 14:17:21.308 [       job-0] INFO  JobContainer         - 
Job start  at             : 2025-10-05 14:17:17
Job end    at             : 2025-10-05 14:17:21
Job took secs             :                  3s
Average   bps             :               67B/s
Average   rps             :              0rec/s
Number of rec             :                   1
Failed record             :                   0

@wgzhao
[feature][plugin][mongodbreader] Support Nested Field Extraction in M…
1522d22 
…ongoDBReader Plugin via Dot Notation

- Enhanced MongoRowConverter to support extracting nested fields from BSON documents using dot notation in the column configuration (e.g., "col1.subcol2", "col1.subcol3.subsubcol4").
- Added getNestedValue method to recursively resolve nested field paths.
- Updated processOne to use getNestedValue for all column extraction, ensuring compatibility with both flat and nested fields.
- Improved code comments for clarity and consistency in English.
- This change allows users to flexibly specify and extract deeply nested MongoDB fields in their job configuration.
@wgzhao wgzhao mentioned this pull request Oct 5, 2025
Open
@wgzhao
wgzhao merged commit b54b542 into master Oct 5, 2025
2 checks passed
@wgzhao
wgzhao deleted the add_nest_fields branch September 4, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@wgzhao