blockSize - a new config paramter for max size of the block by navinrathore · Pull Request #272 · zinggAI/zingg

navinrathore · 2022-05-18T11:59:37Z

No description provided.

sonalgoyal · 2022-05-19T14:55:30Z

 	}
-
+
+	public long getblockSize() {


camel casing is missing - getBlockSize, same for setter

this will break the json parsing - please also test one end to end case with 120l records and set block size in args. Use debug logs to verify

Changed to Camel Case.
Ran with examples/febrl120k/config.json. Output attached. Appropriate block size was selected.

sonalgoyal · 2022-05-19T14:57:36Z

-	public static long getMaxBlockSize(long totalCount) {
+	public static final long MIN_SIZE = 8L;
+	public static long getMaxBlockSize(long totalCount, long blockSizeFromConfig) {
 		long maxSize = 8;


shouldnt this be set to the min size var ?

Used MIN_SIZE

sonalgoyal · 2022-05-19T14:59:46Z

-			if (maxSize > 100) maxSize = 100;
+			LOG.debug("**Block size found **");
+			if (maxSize > blockSizeFromConfig) maxSize = blockSizeFromConfig;
 			if (maxSize <= 8) maxSize = 8;


use the defined constant

used MIN_SIZE

navinrathore · 2022-05-23T09:31:21Z

Output:
examples/febrl120k/config.json

A. ==================================================
Phase: findTrainingData
blockSize(Config): 150

 2022-05-23 12:38:27,107 [main] INFO  zingg.util.Heuristics - **Block size **12 and total count was 12123
 2022-05-23 12:38:27,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 12

B ==================================================
Phase: findTrainingData
blockSize(Config): 10

 2022-05-23 13:56:30,606 [main] WARN  zingg.TrainingDataFinder - Read training samples 37 neg 43
 2022-05-23 13:56:31,093 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - **Block size **10 and total count was 12035
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 10
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 10
 2022-05-23 13:57:00,114 [main] WARN  org.apache.spark.sql.execution.CacheManager - Asked to cache already cached
 
 C ==================================================
Phase: findTrainingData
blockSize(Config): 5
 
  2022-05-23 13:59:39,839 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 11993
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 
D ==================================================
Phase: Train
blockSize(Config): 5

 2022-05-23 14:03:44,657 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 30406
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing file

E ======================================================
Phase: Train
blockSize(Config): 50

ingType,true))]
 2022-05-23 14:08:30,628 [main] INFO  zingg.util.Heuristics - **Block size **30 and total count was 30186
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.Heuristics - Heuristics suggest 30
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 30
 2022-05-23 14:08:43,103 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:08:43,104 [main] WARN  zingg.util.PipeUtil - Writing file
F ======================================================
Phase: Train
blockSize(Config): 25

 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - **Block size **25 and total count was 30061
 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - Heuristics suggest 25
 2022-05-23 14:11:17,583 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 25
======================================================

navinrathore · 2022-05-23T09:55:29Z

Output:
examples/febrl120k/config.json

A. ==================================================
Phase: findTrainingData
blockSize(Config): 150

 2022-05-23 12:38:27,107 [main] INFO  zingg.util.Heuristics - **Block size **12 and total count was 12123
 2022-05-23 12:38:27,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 12

B ==================================================
Phase: findTrainingData
blockSize(Config): 10

 2022-05-23 13:56:30,606 [main] WARN  zingg.TrainingDataFinder - Read training samples 37 neg 43
 2022-05-23 13:56:31,093 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - **Block size **10 and total count was 12035
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.Heuristics - Heuristics suggest 10
 2022-05-23 13:56:50,121 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 10
 2022-05-23 13:57:00,114 [main] WARN  org.apache.spark.sql.execution.CacheManager - Asked to cache already cached
 
 C ==================================================
Phase: findTrainingData
blockSize(Config): 5
 
  2022-05-23 13:59:39,839 [main] INFO  zingg.TrainingDataFinder - Preprocessing DS for stopWords
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 11993
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 13:59:58,061 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 
D ==================================================
Phase: Train
blockSize(Config): 5

 2022-05-23 14:03:44,657 [main] INFO  zingg.util.Heuristics - **Block size **8 and total count was 30406
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.Heuristics - Heuristics suggest 8
 2022-05-23 14:03:44,658 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 8
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:03:59,095 [main] WARN  zingg.util.PipeUtil - Writing file

E ======================================================
Phase: Train
blockSize(Config): 50

ingType,true))]
 2022-05-23 14:08:30,628 [main] INFO  zingg.util.Heuristics - **Block size **30 and total count was 30186
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.Heuristics - Heuristics suggest 30
 2022-05-23 14:08:30,629 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 30
 2022-05-23 14:08:43,103 [main] WARN  zingg.util.PipeUtil - Writing output Pipe [name=null, format=PARQUET, preprocessors=null, props={location=models/101/model/block/zingg.block}, schema=null]
 2022-05-23 14:08:43,104 [main] WARN  zingg.util.PipeUtil - Writing file
F ======================================================
Phase: Train
blockSize(Config): 25

 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - **Block size **25 and total count was 30061
 2022-05-23 14:11:17,582 [main] INFO  zingg.util.Heuristics - Heuristics suggest 25
 2022-05-23 14:11:17,583 [main] INFO  zingg.util.BlockingTreeUtil - Learning indexing rules for block size 25
======================================================

sonalgoyal · 2022-05-23T09:26:29Z

 			maxSize = (long) (0.001 * totalCount);
-			LOG.debug("**Block size found **" + maxSize);
-			if (maxSize > 100) maxSize = 100;
+			LOG.debug("**Block size found **");


please print max size here

navinrathore added 3 commits May 18, 2022 17:28

blockSize - a new config paramter for max size of the block

0317c7f

var renamed to blockSize

4ecadfb

var renamed to blockSize - adding missed files

dbb5555

sonalgoyal requested changes May 19, 2022

View reviewed changes

Merge branch 'zinggAI:main' into BlockingTreeSize259

88b5533

changed blockSize to camelCase in setter/getter

3df49f1

sonalgoyal requested changes May 23, 2022

View reviewed changes

restored maxSize logging

7c9a327

sonalgoyal merged commit b4d6fee into zinggAI:main May 23, 2022

sonalgoyal mentioned this pull request May 23, 2022

expose blocking tree size as a parameter #259

Closed

navinrathore deleted the BlockingTreeSize259 branch June 1, 2022 04:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blockSize - a new config paramter for max size of the block#272

blockSize - a new config paramter for max size of the block#272
sonalgoyal merged 6 commits into
zinggAI:mainfrom
navinrathore:BlockingTreeSize259

navinrathore commented May 18, 2022

Uh oh!

sonalgoyal May 19, 2022

Uh oh!

sonalgoyal May 19, 2022

Uh oh!

navinrathore May 23, 2022

Uh oh!

sonalgoyal May 19, 2022

Uh oh!

navinrathore May 23, 2022

Uh oh!

sonalgoyal May 19, 2022

Uh oh!

navinrathore May 23, 2022

Uh oh!

navinrathore commented May 23, 2022

Uh oh!

navinrathore commented May 23, 2022

Uh oh!

sonalgoyal May 23, 2022

Uh oh!

navinrathore May 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

navinrathore commented May 18, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navinrathore commented May 23, 2022

Uh oh!

navinrathore commented May 23, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants