While working with large custom category datasets, we observed significant scalability limitations in ndpiReader when using the current text-based category loading (-G).
Test setup
• Total domains: ~7,848,883
• Total categories: 96
• Source: /etc/categories/*
• Environment: Linux
Results
Baseline (no categories loaded)
Total memory allocated: 9.03 MB
Setup Time: 107 ms
Using -G (text-based categories)
Total memory allocated: 892.46 MB
Setup Time: 7834 ms
Observations:
• Extremely high memory consumption (~892 MB)
• Very long initialization time (~7.8 seconds)
Using binary .ndb backend (prototype)
Total memory allocated: 9.06 MB
Setup Time: 1128 ms
Observations:
• Memory usage reduced to near baseline levels
• Startup time significantly improved compared to -G
⸻
Problem summary
The current text-based category loading (-G):
• Does not scale well for large datasets (millions of domains)
• Consumes large amounts of RAM due to in-memory structures (AC, strings, etc.)
• Has high initialization overhead
• Negatively impacts runtime performance
⸻
Proposed solution
Introduce (or support upstream) a binary category database format (.ndb) with:
• Precompiled domain index
• mmap-based loading
• Suffix-based lookup (domain → parent domains)
• Fixed category IDs (aligned with ndpi_protocol_category_t)
Key advantages
• Drastically reduced memory usage (no large in-memory AC structures)
• Faster startup (no parsing or building of runtime structures)
• Improved lookup performance
• Better scalability for NGFW / large deployments
⸻
Additional notes
• The prototype maintains compatibility with existing behavior:
• Built-in categories unchanged
• IP-based classification unchanged (Patricia trees)
• Optional hybrid mode (ndb + legacy fallback)
• Normalization and suffix matching follow a consistent contract between generator and runtime
⸻
Use case
This is particularly relevant for:
• NGFW / DPI systems
• Large-scale URL/category filtering
• Deployments with millions of domains
⸻
Conclusion
The current -G approach works well for small datasets, but does not scale to millions of domains.
The .ndb approach demonstrates:
• ~100x reduction in memory usage
• ~7x faster startup vs -G
• Better runtime performance
We believe this could be a valuable addition to nDPI for high-scale environments.
While working with large custom category datasets, we observed significant scalability limitations in ndpiReader when using the current text-based category loading (-G).
Test setup
• Total domains: ~7,848,883
• Total categories: 96
• Source: /etc/categories/*
• Environment: Linux
Results
Baseline (no categories loaded)
Total memory allocated: 9.03 MB
Setup Time: 107 ms
Using -G (text-based categories)
Total memory allocated: 892.46 MB
Setup Time: 7834 ms
Observations:
• Extremely high memory consumption (~892 MB)
• Very long initialization time (~7.8 seconds)
Using binary .ndb backend (prototype)
Total memory allocated: 9.06 MB
Setup Time: 1128 ms
Observations:
• Memory usage reduced to near baseline levels
• Startup time significantly improved compared to -G
⸻
Problem summary
The current text-based category loading (-G):
• Does not scale well for large datasets (millions of domains)
• Consumes large amounts of RAM due to in-memory structures (AC, strings, etc.)
• Has high initialization overhead
• Negatively impacts runtime performance
⸻
Proposed solution
Introduce (or support upstream) a binary category database format (.ndb) with:
• Precompiled domain index
• mmap-based loading
• Suffix-based lookup (domain → parent domains)
• Fixed category IDs (aligned with ndpi_protocol_category_t)
Key advantages
• Drastically reduced memory usage (no large in-memory AC structures)
• Faster startup (no parsing or building of runtime structures)
• Improved lookup performance
• Better scalability for NGFW / large deployments
⸻
Additional notes
• The prototype maintains compatibility with existing behavior:
• Built-in categories unchanged
• IP-based classification unchanged (Patricia trees)
• Optional hybrid mode (ndb + legacy fallback)
• Normalization and suffix matching follow a consistent contract between generator and runtime
⸻
Use case
This is particularly relevant for:
• NGFW / DPI systems
• Large-scale URL/category filtering
• Deployments with millions of domains
⸻
Conclusion
The current -G approach works well for small datasets, but does not scale to millions of domains.
The .ndb approach demonstrates:
• ~100x reduction in memory usage
• ~7x faster startup vs -G
• Better runtime performance
We believe this could be a valuable addition to nDPI for high-scale environments.