[PINOT-4325] Fix auto-correction logic for consecutive failing segmen…

…ts (apache#956)

* [PINOT-4325] Fix auto-correction logic for consecutive failing segments

Modified createConsumingSegment() to consider whether the previous segment was completed or not, when it
tries to create a new consuming segment. If it was not completed, then the new segment is created with the
same start offset as the prev segment (or earliest available, if that is higher than the start offset
of prev segment).

If the prev segment is completed, then we attempt to create the new segment with a start offset equal to
the end offset (unless of course, that offset is not available in kafka, in which we case we pick the
earliest available offset).

In either case, a warning is printed if data is lost, and a table metric is incremented.

Added the setting of Long.MAX_VALUE as the end offset for auto-created segments. This part was missed before.

Added unit tests.

* Add a metric for the auto-creation of segments

Fix the method to search for file paths across different segment form…

…ats (apache#954)

[PINOT-4312] Fix NPE in ValidationManager (apache#952)

Moved the instantiation of ValidationManager to to be after that of PinotLLCRealtimeSegmentManager
so that it can get a handle to the right instance of the latter.

Added an integration test to make sure that a ValidationManager run fixes offline partitions
Conflicts:
	pinot-controller/src/main/java/com/linkedin/pinot/controller/ControllerStarter.java

Fix the method to search for file paths across different segment form…

…ats (apache#954)

[Pinot-Controller] Fixing the segment delete method so that if the de…

…leted segment already exists, we overwrite it. Previously, it would throw a FileExistsException. (apache#931)

Te entity editor improvements and daily report scheduler (apache#920)

* adding improvements in json entity editor

* adding report driver

No-dictionary indexing support for single-valued string columns. (apa…

…che#899)

1. Added new Reader/Writer for variable byte length (string) data
   with customizable compression.

2. Added capability to generate/load segments with columns without dictionary:
   - Forward index contains actual values of data, instead of dictionary ids.
   - For high cardinality usecases, with large variance in string lengths,
     this provides > 2X storage space saving (mostly from eliminating paddingy).
   - Also if data is arranged such that related rows are colocated, this
     improves locality. We have observed orders of magnitued of
improvement in throughput, when data is large enough not to fit in main
memory, as it avoids data fetch from random parts of
dictioary.

3. Added convertor from dictionary encoded to no-dictionary segment.

4. Added unit tests for index creator, as well as var-byte reader/writers.

Fix time boundary sometimes not being computed

Fix the time boundary not being computed if the offline table routing
table is computed before the realtime routing table.

[PINOT-3788] Stop consuming when realtime segment is full (apache#906)

* [PINOT-3788] Stop consuming when realtime segment is full

This condition should happen rarely, if any. If it does happen, we will stop
consuming, indicate to the controller that we have stopped consuming, and go into
ERROR state.

* Addressed review comment

Add console output for all Pinot services (apache#491)

Add console output for the controller, broker and server, along with
port and status information when start up is completed. Display
warnings and errors on the console with colored output.

Reviewer: Adwait