- News Aggregator Apps
- Message Aggregator App Yahoo
- Message Aggregator App Web
- Conservative News Aggregator App
- Message Aggregator App Download
In this post, I'll share a Kafka streams Java app that listens on an input topic, aggregates using a session window to group by message, and output to another topic. This working example could be helpful to find the most frequent log entries over a certain time period.
News Aggregator Apps
Franz (Windows, Mac, Linux) All-in-one messaging app for WhatsApp, Slack, Skype, and many more. One messaging app (aggregator) to rule them all? I do a lot of messaging on a lot of different apps and it gets pretty annoying constantly switching between 6+apps. Not to mention all the documents and attachments that only exist within separate apps.
I used gradle as the build tool and for dependency management. I created a new project via: gradle init.
I added the dependencies to the build file: kafka-log-aggregator/build.gradle
I created a class to represent a log entry consisting of a code, message, and the aggegate count. It uses Gson to deserialize and serialize to json. new file: kafka-log-aggregator/src/main/java/LogEntry.java
Next I create the LogAggregator class which will be used by the Kafka streams app to contain and aggregate all the log entries, file: kafka-log-aggregator/src/main/java/LogAggregator.java
The LogAggregator class has to implement a serializer class to convert to a byte array. file: kafka-log-aggregator/src/main/java/LogAggregatorSerializer.java
And here is the class to deserialize from the byte array. file: kafka-log-aggregator/src/main/java/LogAggregatorDeserializer.java
Next I created the main class to build and run the log aggregator kafka streams app. file: kafka-log-aggregator/src/main/java/LogAggregatorApp.java
I added a scala unit test to ensure the aggregation of logs works as planned. file: kafka-log-aggregator/src/test/scala/LogAggregatorAppTest.scala
I create a simple Kafka producer ruby script to pipe messages onto the topic, wait a while (in this case a minute for the next session window), and pipe some more. file: kafka-log-aggregator/ruby/producer.rb
Message Aggregator App Yahoo
At this point I was ready to start Zookeeper, Kafka, and build/run the streams app:
Message Aggregator App Web
I develop on a Mac using Brew or Docker, here is my environment for this post:
Conservative News Aggregator App
v2.3.0-rc2 Message_Aggregator
Message Aggregator App Download
#18774 Hi, Running the rc2 Message_Aggregator app I don't get any decodes showing in the MA message window. There are no error messages. Everything else is working e.g. typing in the fields I can set free text, change mode, generate standard messages. I can erase Band Activity and Rx Windows, set Calls of Interest and see callsign highlighting in the Band Activity window and the Mode field background changes to cyan when decodes are happening. So UDP communication is ok, just no decodes in the message window. I get the same result running on Linux Mint 19.3 and Ubuntu 20.04. I've tried unicast on the same PC, and multicast on the same PC and over the lan, all with the same negative result. 1st attachment shows WSJT-X and Message_Aggregator v2.3.0-rc2 on Linux Mint 19.3 using unicast - no decodes in MA. Unicast is on 127.0.0.1:2237, multicast on 239.255.0.1:2237 or 239.255.0.2:2237. My nif for the lan enumerates as enp7s0. The firewalls are off and ifconfig shows multicast is enabled. Wireshark shows decode and other datagrams on the local loop i/f when doing unicast and the lan enp7s0 i/f when doing multicast. I re-installed v2.2.2 and did the same tests which worked fine every time, MA messages matching WSJT-X decodes exactly. Also running WSJT-X V2.3.0-rc2 on one PC and Message_Aggregator v2.2.2 on the shack laptop works fine too. see 2nd attachment. I can see there have been changes to the Message_Aggregator app for v2.3.0 e.g. network interface for multicast, but I checked all IP addresses, ports and interfaces match ok. Maybe I'm missing something in the set up of rc2 Message Aggregator, but I can't see it. Any help / advice gratefully received :) 73 Rick (GM4JIB) |
#18775 Hi, Hi Rick, unicast UDP should be no different from previous versions. There are differences with multicast UDP, WSJT-X now sends multicast traffic on the loopback interface by default. On Linux you may have to enable multicast on the loopback interface: If you have WSJT-X and the server application running on different hosts on the subnet, and are using multicast UDP, then you will have to adjust WSJT-X to send multicast on the network facing interface (enp7s0) and adjust the server application to join the multicast group address on an interface on the same network. Windows 7 app store games. 73 |
#18776 Hi Bill Thanks for the quick response. Yes, ifconfig shows multicast is not enabled on the loopback i/f. I assumed multicast would be on the lan interface which is enabled. So I've enabled multicast on the loopback i/f but I'm still not getting messages over unicast or multicast. I don't think the loopback status could account for the fact that status and other datagrams are being received and sent, could it? Re your note 'WSJT-X now sends multicast traffic on the loopback interface by default.' Is this in addition to sending multicast traffic on the multicast address? In the multicast case the rc2 client is sending on the lan i/f (enp7s0) and the rc2 server is also on the same PC so also on the same lan i/f, but no decode messages although the client can be controlled from the server. At the moment I can't run the rc2 message aggregator on another machine as the shack laptop is in another building and only has V2.2.2 installed (it is on the same lan though). I may be able to do that tomorrow. It should be simpler to setup the server on the same pc as the client surely. I also have a java udp server, (coding is work in progress) but it does receive and unpack decode datagrams (and others) successfully when the rc2 client is in both multicast and unicast mode, so I'm quite sure the rc2 client setup is not the issue. Wireshark confirms this. Cheers Rick (GM4JIB) |
#18847 Hi Rick, Bill G4WJS. |
#18854 Hi Bill No problem, thanks for letting me know. I managed to investigate the client server negotiation anyway, using MA and Wireshark, that was the object of the exercise. Thanks again for all the good work. 73s Rick (GM4JIB)
|
