Home

elgo is a management tool for Elasticsearch written in GO, it uses the elastic Go client with a few modifications (like adding snapshot restore functionality)

elgo is easy to use, and supports so far a few features that make it a very powerful tool. The basic syntax is very simple: elgo -action <action> -flag <feature flags>

1. Actions:

elgo supports the following actions:

==> create-index - requires at least -i <index name>. Optional -f input body in json format to pass index 
settings
==> remove-index - requires -i <index name>
==> list-indexes - returns a list of all indexes.
==> index-exists - requires -i <index name>
==> index-doc - requires -i <index name> -type <type> -f input json file to be indexed.
==> create-repo - requires -r <repo name> -type <type> -l <location>.
==> remove-repo - required -r <repo name>.
==> snap-create - required -r <repo name> -s <snap name>. 
Optional -i <index name>. * or multiple indexes accepted.
==> snap-delete - required -r <repo name> -s <snap name>
==> snap-restore - requires -r <repo name> -s <snap name>
==> cluster-info - returns cluster information
==> bulk-request - create/index/delete or update using the bulk interface.
-f <json file> required containing the bulk request to be sent to elasticsearch.
==> search - searches through  indexed documents for a specific field, 
like Id, Type, etc. in one or more or all indexes.

2. create-index

$ ./elgo -action create-index -i elgoindex 
Using Elasticsearch URL:  http://localhost:9200
Successfully created index  elgoindex

Create an index in elastic search. Requires at least [-i indexname ]
If you want to add extra settings, add them as an input file with [-f inputfile].
For example:

./elgo -action create-index -i elgoindex -f i.json 
Using Elasticsearch URL:  http://localhost:9200
Successfully created index  elgoindex
$ cat i.json 
{
  "mappings": {
    "_doc": {
      "properties": {
        "date": {
          "type":   "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        }
      }
    }
  }
}

3. remove-index

requires flag [-i indexname]. Simply deletes an index.

4. list-indexes

No flags, required. Lists all indexes, including system indexes.

5. index-exists

Check whether an index specified with [-i indexname] exists or not. Requires [-i]

6. index-doc

Requires [-i indexname -type type -f inputfile]
Example:

./elgo -action index-doc -i test -type doc -f o.json 
Using Elasticsearch URL:  http://localhost:9200
Indexed document with ID: v0Wfi2cBRQofFjB8q2xb to index: test, type: doc

7. create-repo and remove-repo

Create a repository to hold snapshots. Create repo example:

$ ./elgo -action create-repo                
Using Elasticsearch URL:  http://localhost:9200
Type, name and location are all required for create-repo. [-type type -r reponame -l repopath]
$ ./elgo -action create-repo -type fs -r repo0 -l tmp
Using Elasticsearch URL:  http://localhost:9200
Successfully created repo: repo0 of type: fs at location: tmp

The location specified here with -l will be appended to the path.repo specified in elasticsearch.yml.
If that value is incorrect, elgo will not create a repository and will error out.
To remove a repository:

$ ./elgo -action remove-repo -r repo0                
Using Elasticsearch URL:  http://localhost:9200
Successfully removed repo: repo0.

8. snap-create , snap-delete and snap-restore

snap-create requires at least [-r reponame -s snapname] and optionally, [-i indexpattern].
Index pattern can be an index, a wildcard or "_all" for all, or multiple indexes separated by comma (no spaces). Examples:

$ ./elgo -action snap-create -r repo0 -s snap0 -i _all

$ ./elgo -action snap-create -r repo0 -s snap0 -i ovi,log*,.kibana*

snap-restore works the same way. You can specify an index pattern to control which indexes are being restored, or no index to restore all indexes that are in this snapshot.
snap-restore requires at least [-r reponame -s snapname].

$ ./elgo -action snap-restore -r repo0 -s snap0 -i _all

or to restore only indexes that match the "ovi*" pattern:

$ ./elgo -action snap-restore -r repo0 -s snap0 -i ovi*

It should be noted that if an index that should be restored is open, elgo will error out

snap-delete simply requires [-r reponame -s snapname] to remove a snapshot.

7. cluster-info return basic Elasticsearch cluster information:

$ ./elgo -action cluster-info                          
Using Elasticsearch URL:  http://localhost:9200

    ClusterName: elk-cluster
    Status: yellow
    Number Of Nodes: 1
    Number Of Data Nodes: 1
    Active Primary Shards: 25
    Active Shards: 25
    Relocating Shards: 0
    Initializing Shards: 0
    Unassigned Shards: 20
    Delayed Unassigned Shards: 0
    Number Of Pending Tasks: 0 
    Number Of InFlight Fetch: 0
    Task MaxWait Time In Queue In Millis: 0
    Active Shards Percent As Number: 55.6

8. bulk-request - create/index/delete or update documents using the bulk interface.

This repository contains a few examples of input files that can be used to index/create/update or delete a huge number of documents at once.
The total number of documents that are supported depends on the number of nodes, memory, resources etc. On a simple 2 nodes cluster we could test a few millions.

9. search

The search action requires [-sf searchfield -sv searchvalue] and optionally [-i indexpattern -mr int]
examples:

searches in index "journal", the field "source" for value "journalctl" and prints out 10000 results.

./elgo -action search  -sf source -sv journalctl -mr 10000 -i journal

This search is executed in all indexes, and returns 10 (default) results for "reboot" in all "message" fields.

$ ./elgo -action search  -sf message -sv reboot
Using Elasticsearch URL:  http://localhost:9200
Search took 52 milliseconds.
Found 27 results. 
Document ID:  d0MPi2cBRQofFjB8cqPY
Document Type:  log
message :  -- Reboot --
date :  12-07-2018-18:40:47
source :  journalctl
.
.
.
.
 up to 10 results

Look for messages containing "failed" in indexes journal and matching .kiba* and print one.

$ ./elgo -action search  -sf message -sv failed -mr 1 -i journal,.kiba*
Using Elasticsearch URL:  http://localhost:9200
Search took 1 milliseconds.
Found 2422 results. 
Document ID:  10QPi2cBRQofFjB8cjLk
Document Type:  log
message :  Mar 30 10:02:29 archlinux org_kde_powerdevil[7530]: powerdevil: org.kde.powerdevil.backlighthelper.brightness failed
date :  12-07-2018-18:42:09
source :  journalctl

Searching for panic messages in the logs for example:

./elgo -action search -sf message -sv dump_stack
Using Elasticsearch URL:  http://localhost:9200
Search took 5 milliseconds.
Found 10 results. 
----------------------
Document ID:  dzPwi2cBbp4RVxHyySKw
Document Type:  log
message :  Feb 27 19:26:52 archlinux kernel: [<ffffffff812fde10>] dump_stack+0x63/0x83
date :  12-07-2018-18:41:07
source :  journalctl
----------------------

10.Logging

elgo logs info and errors in a file elgo.out that will be created in the current working directory. Example:

2018/12/07 21:30:20 Successfully created snap snap0 in repo repo0.
2018/12/07 21:34:24 elastic: Error 500 (Internal Server Error): [repo0:snap0/u4-P_i0XTLymEOarXAAKNA] cannot restore index [.monitoring-es-6-2018.12.08] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name [type=snapshot_restore_exception]
....
2018/12/07 22:03:28 Running search on index:  , search field: message, search value: error
2018/12/07 22:03:28 Search took 7 milliseconds.
2018/12/07 22:03:28 Found 9945 results.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly