Repository navigation
[META] Native support for Java plugins #9215
Description
Activity
Hi,
Sorry for the lake of knowledge. I need to write a Java input and output for logstash. I wonder if writing it directly with the new java plugin api.I will need it to production in 4-5 months.
Where can I find documentation about this new java plugin?
When it should be in production ready stage?
Thanks a lot!
In addition I would to contribute from my knowledge to convert some of the plugin from ruby to java.
Thanks
Ziv
@zivziv77, the current plan for this work is to release the Java Plugin API in a series of phases. The first phase is the experimental phase in which we'll solicit feedback on its basic functionality. After that, it would progress to the beta phase where we'd evaluate its stability and then it would be released as GA. I would hope that we would have something within your target timeframe, but I cannot commit to any specific release date.
Right now, that work is happening in the
java-apibranch of the Logstash repository. You can see the current state of the Java Plugin API in theorg.logstash.plugins.apipackage here.It may also be worth mentioning that we do already have a number of hybrid Ruby/Java plugins in which the bulk of the code is Java using Ruby shims to interface with the Ruby plugin API in Logstash. The TCP, date, and DLQ plugins are examples of hybrid Ruby/Java plugins if you'd like some examples of how those work.
@zivziv77, I should also add that we'd be happy to have any contributions you might make in converting plugins from Ruby to Java!
Hi
Thanks for your detailed answer.
The main plugin that I need is Input Redis with cluster support with one of the java Redis client. I thought writing it myself and give it to you also.Where can I find documentation about how the java api works?
I just think it doesn’t make sense writing hybrid plugin java and ruby if there’s going to be support for full java plugin without ruby.
Thanks
@zivziv77, we don't have docs for the Java API, yet. It's pretty simple, so perhaps a brief explanation will provide what you need. (Note that these APIs are subject to change, though it is unlikely that they will change significantly.)
Your Redis input will need to implement the
Inputinterface. Within thestartmethod, you should create a loop that retrieves messages from Redis and writes them to the suppliedQueueWriterinstance as aMap<String, Object>that will be later converted to anEventwithin the Logstash event processing pipeline. In yourstopmethod, you should disconnect from Redis and perform whatever cleanup actions might be required. You can see a very simple example of this basic pattern in theStreamInputclass in that same file.You will also need to provide a constructor that takes two parameters of type
ConfigurationandContext. TheConfigurationinstance should provide your input with configuration values such as the connection parameters for your Redis cluster. You probably won't need to do anything with the providedContextinstance.Let me know if that helps or if you have additional questions.
Thank you!
That sounds pretty straight forward.
One question I didn’t notice where should I register my plugin to be loaded ?
How can I work with logstash on the java api nad mark to load for example Redis-input from java and not from ruby?Oh, right. You'll need to add the
@LogstashPlugin(name = "myRedisPluginName")annotation to your class. If you then addinput { myRedisPluginName {} }(or whatever you call it) to your Logstash config, it will run your Java plugin. Note that you will need to do this with Logstash built from thejava-apibranch as that code has not yet been merged into themasterbranch.I wanted to try to create a filter for two of my (Java) projects (Yauaa and Logparser) so I decided to start by simply trying to build the java filter example.
Apparently the current documentation (yes I know, experimental) assumes more inside knowledge than I currently have.
So the first attempt failed miserably:* What went wrong: A problem occurred evaluating root project 'logstash-filter-java_filter_example'. > Could not get unknown property 'LOGSTASH_CORE_PATH' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.So I download logstash 6.6.0 and hardcoded the path into the build.gradle file.
Now the build runs a bit better but fails with lots of errors indicating it cannot find anything it needserror: package co.elastic.logstash.api does not exist error: cannot find symbol import org.logstash.Event;It would really help me if you can point me a bit more into the right direction (preferably by updating the readme in the example project).
Note that I do not have any experience in Ruby and/or Gradle yet.Thanks.
19 remaining items
Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?
@nielsbasjes, that was an option but a big part of the reason we did not do that is that not all of the public classes in the logstash-core jar are part of the plugin API and are therefore not subject to the backward compatibility guarantees within major versions that we make for the plugin API itself. We did not want to give that impression by publishing a jar that contains nearly all of the Java source code for Logstash.
@YSZYCF, yes, third-party jars are definitely supported. The example Java plugins use those in several places. For example, see the Apache Commons and Log4j2 dependencies in the Java filter:
https://github.com/logstash-plugins/logstash-filter-java_filter_example/blob/master/build.gradle
@danhermann hi.. i am working on a filter plugin in java. I chose to write some part of it in scala classes(for ease of coding) and used that code in java code. Unit tests works fine..configured gradle to compile scala first and then java code.. gem file generated has only one single jar. I was able to install the gem successfully into logstash 7.2 locally.
But while testing the java filter plugin, I noticed, that java part of filter plugin code executes fine, but
scala part of plugin code is not executed at all(I tracked this by setting debug fields into event in java code and scala code)... Do you know what might be going on here? any workarounds are appreciated. Thanks
Java code accesses scala code(ExpressionEvaluator is a scala class) like below and it works fine, in the unit tests that I wrote.
Boolean result = ExpressionEvaluator$.MODULE$.evaluateExpression(event, someString);
also tried accessing scala class like below as well, it didn't help
Boolean result = ExpressionEvaluator.evaluateExpression(event, someString);nevermind, I figured out the issue, it needed scala library packaged into the jar as well, once that included, it worked fine.
@danhermann ，hello，I used the example you supplyed in “https://www.elastic.co/guide/en/logstash/current/java-codec-plugin.html”，and success get the result by the java_stdin。However，when i set input from kafka，I got the error：
here is my config：
Need your help, thank you very much~
@damozhizhou, if you'll post your question in the Logstash forums and tag me (
@danhermann) on it, I'll reply to you there.Closing this as Java plugins went GA in Logstash 7.2.0. Any additional work on Java plugins will be tracked in new issues.
For help with developing your own Java plugins for Logstash, please post in the Logstash forums. You may tag me (
@danhermann) on your post there and I'll try to respond. If you've encountered a bug related to Java plugins, you can create a GitHub issue here.@danhermann
Congratulations this is now GA.
What is the issue I can track about stripping and deploying the API library to maven?The current process cannot be ran in a CI environment because of this missing jar.
@nielsbasjes, could you open an issue here so we can track that effort separately?
Hi,
I built a custom JAVA Input plugin using the example JAVA plugin. I am able to install the GEM in the Logstash in my local. When I tried to install the custom plugin -the generated GEM file in my Dev server, getting the below exception,
Received fatal alert: handshake_failure
Could anyone brief on the process of installing the plugin developed in one machine in another Logstash instance.
Thanks,
KarthikDeivaHave a look at how I did it here https://www.github.com/nielsbasjes/yauaa/tree/master/udfs/logstash/logstash-filter/src/test/docker
I simply install everything in a docker image and check if the output meets my expectations.
This tracks the phases of the Native support for Java plugins Project.
Experimental phase:
Beta phase (#10232):
encodeanddecodemethods?flushmethod for Java filtersco.elastic.logstash.apiGA (#10620):
Beyond GA?