Unit Testing vs. Integration Testing

I was talking with a colleague on the way home, when we started talking about his troubles with his bike.

In short, he could turn the pedals like crazy, and the bike was not moving an inch.

That's when I realized that this is a perfect example of the difference between Unit Testing and Integration Testing.

In short:

unit test

write a unit test, to see if we can properly turn the pedals in the proper direction.

Conclusion: Success! Software works fine. In fact it works extremely well. We achieved a performance increase of over 500%! Yay!

integration test

write an integration test, to see if when we turn the pedals, the bicycle will move.

Conclusion: Failure! Pedals turn fine, but the bicycle doesn't move.

It turns out the integration between the component "pedals" and the component "wheels" was busted.

It totally shows that though Unit tests may all pass, it is not a guarantee that your software is actually doing what it should.

Lesson learned.

Abuse of Lambdas

So, sometimes when you have an existing software structure, with lots of domain rules inside, it becomes difficult to change. Especially when there is a lot of cohesion between your classes.

This is the problem of tight coupling, as opposed to loose coupling¹.

I recently encountered this in our software.

And of course, I needed to change something deep inside.

Now, I didn't need to change anything about the domain logic. It was very good and working as intended.

But I needed a change purely for technical reasons. In this case, the domain logic was used in a Batch Processing on Hibernate Entities, and the domain logic was (in this case, not normally) generating a lot of Entities, and they were all persisted in the Session Context, which naturally grew. And then the dirty check and the session closing takes a long time, etc, etc. See reference [2] on why this's bad.

So I was looking for a way to persist entities, without adding them to all the collections with the CascadeType without changing the domain logic too much, and also keep the default behaviour as standard.

I didn't want to use inheritance, as it was a persisted Hibernate Entity.

So, I came up with the following hack:

Quite simply, I can now add behaviour to an Entity when I need it.

For example:

As me and my colleague say "A seemingly innocent first step on a slippery slope to hell."

Of course, this kind of abuse has always been possible, but it is now easier to do than ever (without having to create an entire class for the priviledge).

References

[1] Wikipedia - Loose coupling

https://en.wikipedia.org/wiki/Loose_coupling

[2] Vlad Mihalcea - The best way to do batch processing with JPA and Hibernate

https://vladmihalcea.com/the-best-way-to-do-batch-processing-with-jpa-and-hibernate/

MERGE SQL Statement

I recently had to do a insert on the database, if the records weren't there, and an update if it was.

Performance was a problem.

Colleague of mine had the same problem when using an insert and update statement with an (exists (select 1)) on 2.5 million records.

After 5 minutes, 5000 rows were done. It would have taken 41 hours to fix them all.

After a merge¹ statement, the entire thing was done in 2.5 minutes.

So, I had to get to grips with the merge statement as well. I hadn't used it before.

There's plenty of tutorials on how it works.

Oracle allows a DELETE statement in the MERGE statement, which seems not to be defined in the SQL Standard.

References

[1] Wikipedia - Merge(SQL)

https://en.wikipedia.org/wiki/Merge_(SQL)

[2] Oracle PL/SQL Documentation - MERGE

https://docs.oracle.com/cd/E11882_01/server.112/e41084/statements_9016.htm#SQLRF01606

Proper use of Optional

Using Optional is hard in the beginning.

Especially for first-timers one has a tendency to use the wrong methods.

Take the following sequence of refactoring for example:

First without any kind of Optional:

A first bad attempt at an optional:

It looks like the person just... kind of removed the Optional back to a situation that he knew and was more familiar with.

Let's try this again:

Now this is a good try. It is very common to see the two Optional class methods being used together. isPresent() and get().

But we can do better.

It's often the same: we wish to do something, only when there's a value available.

Like so:

Of course, it's sometimes better to create a method reference:

But I do find this proliferates the number of methods, but it's an acceptable side effect.

We finally arrive at something concise and easy to read, once you get the hang of it.

Antisamy - false positive security issue

Some time ago, I received a security notification from Github regarding a vulnerability^{1 2} in Antisamy.

It took me a while to find out that it was a false positive.

But the issue in [1] did mention that Antisamy seems to be lacking a maintainer. A worthwhile replacement was suggested as being HTML Sanitizer^{3 5 6}.

I started using the HTML Sanitizer in my project and I like it. The fact that I don't need an XML configuration file is a plus.

There's a StackOverflow answer on why and how of the two projects⁴.

Only thing missing are the policy configuration files from AntiSamy, which are not present in any way in HTML Sanitizer, so you either have to write your own or get them from here⁷.

The file is called HTMLSanitizerAntiSamyPolicy.java.

References

[1] Github - Antisamy issues

https://github.com/nahsra/antisamy/issues/32

[2] CVE details - CVE-2018-1000643

https://www.cvedetails.com/cve/CVE-2018-1000643/

[3] OWASP Java HTML Sanitizer Project

https://www.owasp.org/index.php/OWASP_Java_HTML_Sanitizer_Project

[4] Java: Owasp AntiSamy vs Owasp-java-html-sanitize

https://stackoverflow.com/questions/28577738/java-owasp-antisamy-vs-owasp-java-html-sanitize/29259874

[5] GitHub - OWASP/java-html-sanitizer

https://github.com/OWASP/java-html-sanitizer

[6] MailingList - HTML Sanitizer

https://lists.owasp.org/pipermail/owasp-leaders/2011-March/004883.html

[7] GitHub - ESAPI/esapi-java-legacy - New file to use Java HTML Sanitizer that implements old AntiSamy poli…

https://github.com/ESAPI/esapi-java-legacy/commit/d48e5a6f07601322c44c113058526eb133b777a5#diff-72ca1caf39f169db1ab83d2af2ec6cc3l