-
Notifications
You must be signed in to change notification settings - Fork 753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Analyzer plugin framework #2660
base: master
Are you sure you want to change the base?
Conversation
2022f6d
to
2fc9f32
Compare
2fc9f32
to
c19d0f2
Compare
c19d0f2
to
bd0fae0
Compare
opengrok-indexer/src/main/java/org/opengrok/indexer/framework/PluginClassLoader.java
Show resolved
Hide resolved
@vladak for review |
fc2c391
to
b27c0d0
Compare
*/ | ||
package org.opengrok.indexer.analysis; | ||
|
||
public interface IAnalyzerPlugin { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please delete the I
prefix? I know it is used in C# world but I don't see it that often in Java and I read in some books it should not be used because it shows unnecessary implementation details.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No doubt for that. The trouble here is that we’ve already exposed one interface for plugins with I so this is consistent with the current code.
We should make a change in a future to transfer both cases to the correct name but it’d be a breaking change and I think it shouldn’t be part of this pr.
opengrok-indexer/src/main/java/org/opengrok/indexer/analysis/AnalyzerFramework.java
Outdated
Show resolved
Hide resolved
private static void registerAnalyzer(AnalyzersInfo analyzersInfo, AnalyzerFactory factory) { | ||
for (String name : factory.getFileNames()) { | ||
AnalyzerFactory old = analyzersInfo.fileNames.put(name, factory); | ||
assert old == null : |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, it's been so long I saw assertions :D Maybe last time in Lucene code.
Do we run the code with assertions enabled? I personally don't.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a very old code and should be refactored. No production code should ever rely on assert statements.
* @throws IllegalAccessException if the constructor cannot be accessed | ||
* @throws InstantiationException if the class cannot be instantiated | ||
* @throws NoSuchMethodException if no-argument constructor could not be found | ||
* @throws ClassCastException if the class is not a subclass of {@code |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are you aligning the description? I personally don't like it because if I were to add VeryLongLongLongLongException
then I would have to modify other places that have nothing to do with my changes which just makes the diff hard to read. That's the reason we don't do following:
int abc = 1;
int a = 2;
int ab = 3;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Idea does that for all of javadoc stuff.
Descriptions and the actual code are two different things.
* {@link AnalyzerFramework#addPrefix(String, AnalyzerFactory)} | ||
* are called to augment the value in {@link AnalyzerGuru#getVersionNo()}. | ||
*/ | ||
public final SortedSet<String> customizations; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please add getters? It just does't feel right this way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I intentionally don’t want to use getters as this class does no provide any functionality so final properties are enough
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even simple DTO classes are always written with getters/setters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’ve seen codes which don’t follow that. If this is code style for opengrok I can change it of course.
opengrok-indexer/src/test/java/org/opengrok/indexer/analysis/AnalyzerFrameworkTest.java
Show resolved
Hide resolved
opengrok-indexer/src/main/java/org/opengrok/indexer/analysis/AnalyzerFramework.java
Outdated
Show resolved
Hide resolved
* @param b the second instance | ||
* @return true if we consider them different, false otherwise | ||
*/ | ||
private static boolean factoriesDifferent(AnalyzerFactory a, AnalyzerFactory b) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should this class be Comparable
then ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an old code, I can do this in another PR so I don't mix refactoring with actual changes.
*/ | ||
public synchronized AnalyzerGuru getAnalyzerGuru() { | ||
if (analyzerGuru == null) { | ||
analyzerGuru = new AnalyzerGuru(getPluginDirectory()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should just split these 2 plugin directories into analyzers and authorization plugins. This way it is kind of confusing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not necessary as it's called plugins directory and you can put all your plugins there be it analyzer or authorization plugin. The application frameworks will load only the plugins they support.
However, if it's desired I can split these.
I wonder how does this interacts with analysis performed from within the webapp. |
It would be nice if you could produce dummy analyzer plugin, ideally using language other than Java. That would also serve as a basis for documentation. |
*/ | ||
public static final AnalyzerFactory DEFAULT_ANALYZER_FACTORY = new FileAnalyzerFactory(); | ||
|
||
private static final IAnalyzerPlugin[] DEFAULT_PLUGINS = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume this is a temporary measure until the framework is fully implemented and in the final state it will dynamically load all the plugins from a directory - no hard-coding needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes this is for seamless transition from the old approach to the new approach.
That is a good point though. I'm affraid that with that approach we would need to distribute two jar files instead of one and more to force people to actually put the second one somewhere (plugins directory) and point the configuration directory there and I think this is a little bit more than what I'd expect to get opengrok working.
This is beyond my knowledge, maybe @tarzanek can produce the pdf analyzer to test this |
What do you mean? |
- loading analyzers implementing an interface from a directory - augmenting analyzer guru to use the loaded plugins - making analyzer guru a property of RuntimeEnvironment - loading the default analyzers as it used to be approaches oracle#2588
b27c0d0
to
6dd520c
Compare
Rebased and updated. Make it a priority for next comments as any change in analyzer guru makes this hard to rebase correctly. |
I did a unit test with a dummy analyzer to test the basic functionality. |
c94e896
to
4853187
Compare
Please hold on merging. I have some questions and issues and will write them up from work later today. |
I do like this patch's refactoring the static The approach of this patch treats OpenGrok's supported analyzers as merely a collection of fully independent While OpenGrok can match filename extensions or prefixes, it also selects analyzers in order-dependent non-heuristic and heuristic stages, relying on You write that your approach provides "seamless transition from the old approach to the new approach" but really the new approach would not support properly registering the existing set of analyzers; the application does not break only because you copy the static ordering for For supporting @tarzanek's desire for a PDF plug-in, only a plug-in analyzer that keys off of file extension would succeed reliably. Any implemented For another example of how misinterpreting OpenGrok's supported analyzers as fully independent does not accomodate extensibility as more analyzers are incorporated: to fully support Objective-C, it is not enough to add an Objective-C analyzer and factory. This patch is lacking any approach to modify @vladak in #2588 directs that with an analyzer plug-in framework it "should be possible to add options to Ctags," which I agree is a necessary part of a pull request named "Analyzer plug-in framework" that is not just a trivial class loader. Please may I head off any suggestion that this patch is but an introductory stage of an analyzer framework by remarking that this patch in its form is just a trivial jar/class loader [unlike your nice, stack-preserving Perhaps we should work to revise the order-dependent Overall I think some effort should be made to tackle some of the API improvements before adding jar/class loading. Otherwise we may find ourselves in a difficult architectural position with permanent, limited support of meaningful plug-ins.
You bring up the difficulty of rebasing large branches, which vexes me too. May I ask why you casually reordered a number of methods in BTW there are a couple of regressions and a log message error in this patch which I did not line-comment because of high-level concerns which I wanted to bring up first. In my opinion that IDEA style of spaced-out JavaDoc should not be accepted by us. Lastly I am reluctantly forced to name you as a serial offender in lifting code to different files without complying with CDDL's requirements for maintaining in the new files the copyright and descriptive text giving attribution for the original source. |
This is exactly what this is meant to be. Transition from the old state to the new state with as minimal number of changes so it can be reviewed easily rather than fat PR with mixed refactoring and features. |
Yes I thought you might discount everything I brought up in my review by claiming a jar/class loader as a meaningful initial stage when the forgone API changes are more significant in my opinion. As I wrote, "I think some effort should be made to tackle some of the API improvements before adding jar/class loading." |
Will change, why not use an inline comment?
No problem I'll change it, why not use an inline comment?
As it's in the description it only approaches the plugin framework - only the loading part. This is not to be meant as a complete solution rather a first step forward.
IDEA -> refactor –> move. I can walk though it again by myself.
I'd like to know which ones as this does not really bring much new functionality nor changes any order for the analyzers.
In my opinion it does not matter. If this is a consensus I can conform to it.
That's why we have reviews. Why not using inline comments for affected files? |
This is not supported now and should be tackled when there is a need for it in my opinion. |
When it's a broad problem, I put it in a top-level review which I don't think should affect your ability as the patch owner to correlate to the source code.
I have raised pull requests related to this and have existing customizations for the same. If we're supporting a plug-in model where folks can customize OpenGrok without sharing their modifications, then my existing use cases are valid too. |
So as I understood the main concern is that we go from order-dependent approach to order-indepenent approach and the new loaded analyzers wouldn't quite work with that. I'll look into that. |
If you or folks already customize their OpenGrok analyzers then I don't think a plug-in model should affect your or folks's ability to customize it in any way further. |
@vladak asked, "I wonder how does this interacts with analysis performed from within the webapp." The webapp can use |
As the webapp loads the same plugins, I wouldn't expect any differencies, however, will verify it. |
*/ | ||
|
||
/* | ||
* Copyright (c) 2019 Oracle and/or its affiliates. All rights reserved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems derived from AnalyzerGuru
so missing attribution.
ver |= (long)customizationHashCode << 32; | ||
final int customizations = Objects.hash(this.pluginFramework.getAnalyzersInfo().customizations.toArray()); | ||
if (customizations != 0) { | ||
ver |= (long) customizations << 32; | ||
} | ||
return ver; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this value should be plug-in-aware
* @throws ForbiddenSymlinkException if symbolic-link checking encounters | ||
* an ineligible link | ||
*/ | ||
public void populateDocument(Document doc, File file, String path, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
* @throws java.io.IOException If an error occurs while accessing the data | ||
* in the input stream. | ||
*/ | ||
public AbstractAnalyzer getAnalyzer(InputStream in, String file) throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
* @throws java.io.IOException If an error occurs while accessing the input | ||
* stream. | ||
*/ | ||
public String getContentType(InputStream in, String file) throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
* @param project Project the file belongs to | ||
* @throws java.io.IOException If an error occurs while creating the output | ||
*/ | ||
public static void writeXref(AnalyzerFactory factory, Reader in, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
* @param project project the file belongs to | ||
* @throws java.io.IOException if an error occurs while creating the output | ||
*/ | ||
public static void writeDumpedXref(String contextPath, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
* @param factory the factory | ||
* @see AnalyzerFramework#addPrefix(String, AnalyzerFactory) | ||
*/ | ||
public void addPrefix(String prefix, AnalyzerFactory factory) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
* @param factory the factory | ||
* @see AnalyzerFramework#addExtension(String, AnalyzerFactory) (String, AnalyzerFactory) | ||
*/ | ||
public void addExtension(String extension, AnalyzerFactory factory) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why relocate?
*/ | ||
|
||
/* | ||
* Copyright (c) 2019 Oracle and/or its affiliates. All rights reserved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing attributions?
@@ -149,8 +149,8 @@ public static void main(String[] argv) { | |||
PrintStream helpStream = status != 0 ? System.err : System.out; | |||
helpStream.println(helpUsage); | |||
if (helpDetailed) { | |||
helpStream.println(AnalyzerGuruHelp.getUsage()); | |||
helpStream.println(ConfigurationHelp.getSamples()); | |||
System.err.println(AnalyzerGuruHelp.getUsage(RuntimeEnvironment.getInstance().getAnalyzerGuru())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
regression
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right
helpStream.println(AnalyzerGuruHelp.getUsage()); | ||
helpStream.println(ConfigurationHelp.getSamples()); | ||
System.err.println(AnalyzerGuruHelp.getUsage(RuntimeEnvironment.getInstance().getAnalyzerGuru())); | ||
System.err.println(ConfigurationHelp.getSamples()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
regression
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right
@@ -216,7 +216,8 @@ private PluginType loadClass(String classname) throws ClassNotFoundException, | |||
|
|||
// check for implemented interfaces or extended superclasses | |||
for (Class intf1 : getSuperclassesAndInterfaces(c)) { | |||
if (intf1.getCanonicalName().equals(classType.getCanonicalName()) | |||
if (intf1.getCanonicalName() != null && // anonymous classes | |||
intf1.getCanonicalName().equals(classType.getCanonicalName()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See line 226 below which logs re IAuthorizationPlugin
that is no longer applicable for all PluginFramework
subclasses
ordering of analyzers is important to fix (note to this issue from Krystof) |
I've thought about possible solutions:
|
Looks like this needs some work (and rebase), converting to draft. |
Introducing an analyzer plugin framework
depends on #2659
fixes #2027
approaches #2588