Scribd
Upload
13 views2 pages

RTMC

The document outlines the development of a Real-time Data Classification API that processes live data streams and classifies them based on user-defined rules while using JWT for authentication. It details a phased approach including project setup, authentication, database integration, classification rule management, real-time data processing, and testing. Engineers are encouraged to utilize ChatGPT-4.0 for assistance and focus on clean code practices and scalability.

Uploaded by

Aftab hussain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
13 views2 pages

RTMC

The document outlines the development of a Real-time Data Classification API that processes live data streams and classifies them based on user-defined rules while using JWT for authentication. It details a phased approach including project setup, authentication, database integration, classification rule management, real-time data processing, and testing. Engineers are encouraged to utilize ChatGPT-4.0 for assistance and focus on clean code practices and scalability.

Uploaded by

Aftab hussain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Project: Real-time Data Classification API

Objective
Build an API that processes a live stream of data, classifies it based on user-defined
classification rules, and authenticates incoming requests using JWT tokens.

Language
● Go (recommended)
● Node.js
● Python

Tools and Resources


• ChatGPT-4.0 for assistance and guidance.
• JWT for authentication.
• MongoDB or PostgreSQL for storing rules and user data.
• WebSocket or Server-Sent Events (SSE) for live data streaming.

Task List
Phase 1: Initial Setup
1. Project Initialization
– Set up the project repository.
– Initialize the project using Go or Node.js.
– Set up environment configurations.
2. Dependency Management
– Install necessary packages/libraries for JWT, database connectivity, and
WebSocket/SSE.
– Set up dependency management (Go modules or npm/yarn).

Phase 2: Authentication
3. JWT Authentication
– Implement JWT authentication middleware.
– Ensure incoming requests contain a valid JWT token in the header.
– Create signup and login endpoints (if they don’t exist).

Phase 3: Data Models and Database


4. Database Schema
– Design and implement database schema for storing user-defined rules.
– Create models for user and classification rules.
5. Database Integration
– Implement database connection and setup.
– Create CRUD operations for user-defined rules.

Phase 4: User-Defined Classification Rules


6. Classification DSL
– Design a Domain Specific Language (DSL) for user-defined classification
rules.
– Implement a parser to interpret the DSL.
7. API for Rule Management
– Implement endpoints to create, read, update, and delete classification rules.
– Ensure rules are validated and constrained by a user-defined maximum.

Phase 5: Real-time Data Processing


8. Data Streaming Integration
– Set up WebSocket or SSE for live data streaming.
– Implement a mechanism to receive and process incoming data streams.
9. Classification Engine
– Develop a classification engine to apply user-defined rules to the incoming
data.
– Ensure the engine operates in real-time with minimal latency.

Phase 6: Testing and Validation


10. Unit and Integration Tests
– Write unit tests for all modules.
– Implement integration tests for end-to-end validation.
11. Load Testing
– Perform load testing to ensure the system can handle high volumes of live
data.

Notes for the Engineers


• Utilize ChatGPT-4o for assistance with any coding or conceptual challenges.
• Focus on clean code practices and robust error handling.
• Ensure the system is scalable and can handle real-time data efficiently.
• Regularly commit and push code to the repository for version control and
collaboration.
Good luck with the project! If you have any questions or need further clarification, feel free
to reach out.

Link -
https://jamboard.google.com/d/1jv5QnEMIa-DcYMu2kgs_-82oVRhkUgnjUEZMX9Keg3s/vi
ewer?f=0

You might also like