I’m interested in trying developing a small plugin to learn how to extended jami user experience. I’ve found the plugin technical information page but it is missing basic information that I did not found in the documentation. Here is a list of newbie questions about plugin development.
What is the required knowledge to develop a new plugin?
C++
Qt development?
Signals
QML
What is recommended documentation to learn these subjects?
The required knowledge to develop a new plugin is C++. All of current plugins are written in that language.
There are a lot of documentation and tutorials in the Internet for someone who wants to learn C/C++. If you already have an experience with programming they should not be too hard to understand. Specifically speaking about the plugins, we have 5 from which you can learn. They can serve as a baseline for your development. Refer to: savoirfairelinux / jami-plugins · GitLab
Each plugin is different from another. You can have a plugin that depends on some libraries and others that don’t. For instance, GreenScreen depends on onnx, opencv and ffmpeg while AutoAnswer does not require any additional dependencies. Apart from that, all plugins will share: clang for Android, gcc for MacOS and Linux and CMake (Visual Studio 16 2019) in Windows.
Plugins preferences are already displayed in the client for GNU/Linux, MacOS, Windows and Android. In the same UI the user is able to change the preference values.
The goal with Jami plugins is to be the most flexible possible. Currently the plugins can interact with text messages and with the audio and video streams available during a call. The support to display custom web views is under development at this moment and will be soon available. If you are to use libraries that are already used in the daemon we strongly recommend to use our contrib version in the daemon repository, otherwise you may feel free to use whichever library is necessary for your functionality.
Plugins have been successfully developed and tested for GNU/Linux, Windows, MacOS and Android platforms. Jami for IOS does not support plugins.
Furthermore, if any of your questions is not yet fully answered, please do elaborate it a bit more.
Also, if you find that something is undocumented, the doc can be modified via any Pull Request to the jami-docs repository.
Thanks for the answers. I’ll try to keep this thread for general questions (and didactic answers) about plugin development and I’ll ask specific questions in other posts.
Question 7 was more about plugin supporting jami features available only on specific platforms. For instance, the full TensorFlow not available on Android. How do me manage the differences?
Can plugins do something other than interacting with stream data, like interacting with the user interface? For instance, is it possible to write a chat plugin to check message wording with the system spell checker before posting in the chat?
Can a misbehaving plugin crash jami-daemon or jami-qt?
Can a plugin use shared/existing features or libraries from jami-daemon or jami-qt? For instance, I see that jami-daemon has a Logger class. Can the plugin just include it to use it or does it need to register to use a common logger service? I used the logger as an example because it is a commonly shared feature, even if I’ve seen there’s a Plog class…
Can we chain plugins? For instance, if I want to use both GreenScreen and CatEars (a fictitious plugin filter that adds cat ears to a people picture).
What is the order plugins access a stream? For instance, if a user wants to use the plugins GreenScreen and WaterMark, it makes sense only if GreenScreen is applied first and WaterMark after. Is it possible to specify the order or plugin priority?
Will you publish the AutoAnswer in the plugins page? People will use it if they can access the JPL file (plugin package), and its code is simple and easy to understand.
*Sorry for future plugin developers, I had to remove URLs as new users can only have 2 links in posts
The Tensorflow restriction is not directly related to Jami. It is rather a TF library characteristic. In the case you have libraries code that behave differently, like TF or ONNX, from one platform to another, the plugin code should embrace and handle that difference. In C/C++ that is most commonly done through a “ifdef” along the code. You might see that in the GreenScreen code (with onnx) and the Tensorflow plugin.
Moreover, the only platform that does not offer plugins is IOS. For GNU/Linux, Windows, MacOS and Android, all plugin system is available without restrictions.
The plugins system currently support audio/video streams and chat messages. For the second, the system allows a message to be changed prior to send it and after receiving. This means that you can create, for instance, a plugin that will perform spells check, or that will translate what you receive to a preferred language. More than that, the plugins can send a message on your behalf or read a received message and perform an action. Image that you want to use Jami to control a device on your home. With the good libraries you might be able to send messages with the text “turn on” or “timer 30 min” and the plugin will receive those messages and act accordingly.
Yes, a plugin can crash Jami.
In theory yes, you can include the files containing the feature you desire from the daemon into your plugin and compile it. However given Jami is a big code, the code part you want to include might request others unneeded files. Depending of what you want to include, you might end by compiling a big sized plugin which we discourage. In the plugins repository you might find some code very similar to what you can find in the daemon, but it is in reality a reduced/simplified version that removes includes to avoid the domino effect.
The plugin system provides an API to the plugins that allows them to call services (or callbacks). For instance, if a plugin wants to send a message, it can call a “sendTextMessage” service with some data and it will be the daemon that will handle the request. You can check a service call in:
and you can see the service creation in the daemon side here:
Please note that the links above are not exhaustive and other examples can be found in the daemon and in the plugins repositories.
Yes we can use multiple plugins at a time.
Specifically for the GreenScreen plugin, it will take effect before all others. So you can use both it and WaterMark and they both will be visible. The same is not valid if you are to use both WaterMark and HelloWorl or XXXX. The plugin system might improve from offering the user the option to set the order in which the plugins are applied.
The request will be passed to the person responsible for jami.net.
The build.sh template of the SDK seems to expect clang++ under linux. Is it a bug? Should it be replaced by gcc?
Also is Debian a supported distribution? The build script can’t be run from the SDK/pluginMainSDK.py and I have to use build-plugin.py --distribution=ubuntu to start building, though with compilation errors (missing fmt/core.h).
I don’t know if I have to open issues in Gitlab when I encounter such problems, as I’m not sure the problem is on my side or not, and the plugins SDK documentation is not really helpful on how all this is setup. For instance, there are no mentions of the --distribution requirement. And how the various build scripts run.
About the clang, it was a honest mistake. At some point it was discussed to change to gcc but we did not implemented it. Your are right and in the build.sh script only clang is used.
The plugins build were never tested under a Debian platform, only with ubuntu and redhat. If you are trying to build within a Debian environment and you encountered issues, either the build logs or a patch is most welcome. Moreover, the available x86_64-linux should work.
If the plugin SDK fails to build, it might be for the same problem you encountered with the build-plugin.py script. To fix that you need to cd <jami-daemon>/contrib/native && ../bootstrap --enable-fmt && make .fmt .
--distribution is only used to build for android, you should not need to set it. The use of that argument follows the same structure as for the daemon build.
After all those doubt points you bring up, it is clear that the plugin documentation needs a revision.