Thursday, 21 January 2016

Yum replaced by DNF - Followup

Well, recently came into the problem that I had some configuration that requires porting, I think. I wanted to get into the DNF scene, whilst Centos 7 still uses yum by default.

From Yum to DNF

I installed dnf, simply using.

yum install dnf

The packages yum, yum-plugin-fastestmirror, yum-utils are now no longer needed. Your original configuration file can then be found in yum.conf.rpmsave.

I installed "dnf-yum", to be able to use yum as a automatic redirect to dnf.

Yum.conf

The /etc/yum.conf shall have been replaced with /etc/dnf/dnf.conf.

Yum cron jobs

In DNF, this is replaced by the package "dnf-automatic"³. The configuration of which is in /etc/dnf/automatic.conf.

Removed package yum-cron as it is no longer needed.

To get started:

systemctl enable dnf-automatic.timer && systemctl start dnf-automatic.timer

To list the timers set:

systemctl list-timers *dnf-* -all

Excluding packages

I have added the java* packages to the exclude directive in yum.conf¹.

Of course, I do wish to manually install the java updates along with all the other updates². To do this I can use the command line:

yum --disableexcludes=all update

These switches in the commandline and changes in the config file of yum are ported identically over to dnf. So that's not a problem.

Yum Repos

Apparently, DNF still uses all the repositories defined under /etc/yum.repos.d. So that's a relief.

References

[1] Redhat - How do I exclude kernel or other packages from getting updated in Red Hat Enterprise Linux while updating system via yum?: https://access.redhat.com/solutions/10185
[2] nixCraft - Force yum update Command To Exclude Certain Packages: http://www.cyberciti.biz/faq/redhat-centos-linux-yum-update-exclude-packages/
[3] Fedora Project - AutoUpdates: https://fedoraproject.org/wiki/AutoUpdates

Thursday, 14 January 2016

Glassfish Logging to Database

I wanted to redirect the logging messages from the Glassfish Application Server to the database.

Glassfish uses the standard Logging API available in the JDK. There are some who find the already existing logging APIs (log4j, etc) available better than the default provided in the SDK. But I find it sufficient for my purposes.

The Logging API uses Handlers to indicate what needs to be done with all those log messages. The default handlers available in the API are shown in the diagram below.

I have opted for having the logrecords sent via the network (local interface) to a small daemon that posts the log records into a database table. So that would mean using the SocketHandler.

In general, this is what you want. Several application servers all sending their log messages to a central repository that can deal with it. Of course, there are way more sophisticated solutions readily available compared to this home-brewed thing. But it helps to get the general idea of how it works.

The software I made is available at https://github.com/maartenl/sql-logging. The sequence diagram below is basically how it works.

Since the data send over the network is in the XML format, I'm using StAX for interpreting the XML stream properly.

The funny thing is that the DatabaseHandler in the diagram above, is basically a child of the Handler class (depicted in the class diagram up top). So, what we have here is a kind of Logging proxy, really.

The database table follows the fields in the LogRecord class closely:

create table mm_systemlog (
  id bigint(20) NOT NULL AUTO_INCREMENT primary key,
  millis bigint(20) not null,
  sequence bigint(20) not null,
  logger varchar(255),
  level varchar(25),
  class varchar(255),
  method varchar(255),
  thread int(10),
  message text not null,
  INDEX(millis)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Glassfish

Glassfish has a logging.properties file (glassfish4.1/glassfish/domains/domain1/config/logging.properties). It is possible to add or change the Log Handler used by Glassfish by changing it in the logging.properties file^{1 3}.

I was unable to provide Glassfish with a Logging Handler that directly posted the LogRecords into a database (see [2]). Hence the current solution described above.

The (fairly easy) steps were as follows:

the developer has put the custom handler JAR file into the domain-dir/lib/ext directory.
the class that extends java.util.logging.Handler must be in the server classpath.
add the new handler to the handlers attribute in the glassfish/domains/domain1/config/logging.properties file

handlers=java.util.logging.ConsoleHandler,com.mrbear.logging.handlers.SimpleSocketHandler
reboot glassfish

XML Formatter

Below is an example of what the XML format looks like.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE log SYSTEM "logger.dtd">
<log>
<record>
<date>2016-01-11T06:50:33</date>
<millis>1452491433234</millis>
<sequence>0</sequence>
<logger>com.mrbear.logging.tests.SimpleSocketHandlerTest</logger>
<level>SEVERE</level>
<class>com.mrbear.logging.tests.SimpleSocketHandlerTest</class>
<method>testSocketHandler</method>
<thread>1</thread>
<message>Hello, World</message>
</record>
<record>
<date>2016-01-11T06:50:33</date>
<millis>1452491433261</millis>
<sequence>1</sequence>
<logger>com.mrbear.logging.tests.SimpleSocketHandlerTest</logger>
<level>INFO</level>
<class>com.mrbear.logging.tests.SimpleSocketHandlerTest</class>
<method>testSocketHandler</method>
<thread>1</thread>
<message>Welcome Home</message>
</record>
<record>
<date>2016-01-11T06:50:33</date>
<millis>1452491433263</millis>
<sequence>2</sequence>
<logger>com.mrbear.logging.tests.SimpleSocketHandlerTest</logger>
<level>CONFIG</level>
<class>com.mrbear.logging.tests.SimpleSocketHandlerTest</class>
<method>testSocketHandler</method>
<thread>1</thread>
<message>Config ....</message>
</record>
<record>
<date>2016-01-11T06:50:33</date>
<millis>1452491433265</millis>
<sequence>3</sequence>
<logger>com.mrbear.logging.tests.SimpleSocketHandlerTest</logger>
<level>FINE</level>
<class>com.mrbear.logging.tests.SimpleSocketHandlerTest</class>
<method>testSocketHandler</method>
<thread>1</thread>
<message>Fine ....</message>
</record>
<record>
<date>2016-01-11T06:50:33</date>
<millis>1452491433266</millis>
<sequence>4</sequence>
<logger>com.mrbear.logging.tests.SimpleSocketHandlerTest</logger>
<level>FINEST</level>
<class>com.mrbear.logging.tests.SimpleSocketHandlerTest</class>
<method>testSocketHandler</method>
<thread>1</thread>
<message>Finest ....</message>
</record>
<record>
<date>2016-01-11T06:50:33</date>
<millis>1452491433266</millis>
<sequence>5</sequence>
<logger>com.mrbear.logging.tests.SimpleSocketHandlerTest</logger>
<level>WARNING</level>
<class>com.mrbear.logging.tests.SimpleSocketHandlerTest</class>
<method>testSocketHandler</method>
<thread>1</thread>
<message>Warning ....</message>
</record>

I would like to point out that, strictly speaking, the above XML document is not valid, as there is no closing </log> tag.

This makes sense, as the xml is streamed and never quite ended.

I would also like to point out that the XML document contains for each record both the number of milliseconds sinds 1970 and the date. This at first seems superfluous to me, but after some thought it makes sense. If you add both, you get valuable information in what timezone the server logging messages is working. This information is most likely lost in the milliseconds approach.

References

[1] Oracle Blog - Configure my Custom Log Handler in GlassFish 3.1: https://blogs.oracle.com/naman/entry/configure_my_custom_log_handler
[2] StackOverflow - Capture GlassFish log file into SQL/JPA data base: http://stackoverflow.com/questions/12397861/capture-glassfish-log-file-into-sql-jpa-data-base
[3] 7 Administering the Logging Service - Adding a Custom Logging Handler: Oracle GlassFish Server Administration Guide
HK2 - Dependency Injection Kernel: https://hk2.java.net/2.3.0/
Oracle Java^TM Tutorials - Lesson: JDBC Introduction: https://docs.oracle.com/javase/tutorial/jdbc/overview/index.html
JavaBendeR - Simple Log server with java SocketHandler and centralization of log records: http://javabender.blogspot.nl/2010/09/simple-log-server-with-java.html
Java Logging API Tutorial – Examples of Logger Levels, Handlers, Formatters and Filters: http://www.journaldev.com/977/java-logging-api-tutorial-examples-logger-levels-handlers-formatters-filters
Oracle Java^TM Tutorials - Reading XML Data into a DOM: https://docs.oracle.com/javase/tutorial/jaxp/dom/readingXML.html
Oracle Java^TM Tutorials - Using StAX: https://docs.oracle.com/javase/tutorial/jaxp/stax/using.html