Unit Testing vs. Integration Testing

I was talking with a colleague on the way home, when we started talking about his troubles with his bike.

In short, he could turn the pedals like crazy, and the bike was not moving an inch.

That's when I realized that this is a perfect example of the difference between Unit Testing and Integration Testing.

In short:

unit test

write a unit test, to see if we can properly turn the pedals in the proper direction.

Conclusion: Success! Software works fine. In fact it works extremely well. We achieved a performance increase of over 500%! Yay!

integration test

write an integration test, to see if when we turn the pedals, the bicycle will move.

Conclusion: Failure! Pedals turn fine, but the bicycle doesn't move.

It turns out the integration between the component "pedals" and the component "wheels" was busted.

It totally shows that though Unit tests may all pass, it is not a guarantee that your software is actually doing what it should.

Lesson learned.

Abuse of Lambdas

So, sometimes when you have an existing software structure, with lots of domain rules inside, it becomes difficult to change. Especially when there is a lot of cohesion between your classes.

This is the problem of tight coupling, as opposed to loose coupling¹.

I recently encountered this in our software.

And of course, I needed to change something deep inside.

Now, I didn't need to change anything about the domain logic. It was very good and working as intended.

But I needed a change purely for technical reasons. In this case, the domain logic was used in a Batch Processing on Hibernate Entities, and the domain logic was (in this case, not normally) generating a lot of Entities, and they were all persisted in the Session Context, which naturally grew. And then the dirty check and the session closing takes a long time, etc, etc. See reference [2] on why this's bad.

So I was looking for a way to persist entities, without adding them to all the collections with the CascadeType without changing the domain logic too much, and also keep the default behaviour as standard.

I didn't want to use inheritance, as it was a persisted Hibernate Entity.

So, I came up with the following hack:

	@Entity
	public class ConstructionPlan implements Serializable
	{
	@Id
	private Long id;
	public List<ConstructionPermit> constructionPermits;
	@Transient
	private Consumer<ConstructionPermit> consumer;
	public void setConsumer(Consumer<ConstructionPermit> consumer)
	{
	this.consumer = consumer;
	}
	public void addPermit(ConstructionPermit permit)
	{
	if (consumer != null)
	{
	consumer.accept(permit);
	return;
	}
	constructionPermits.add(permit);
	}
	}

view raw ConstructionPlan.java hosted with ❤ by GitHub

Quite simply, I can now add behaviour to an Entity when I need it.

For example:

	ConstructionPlan constructionPlan = new ConstructionPlan();
	constructionPlan.setConsumer(permit ->
	{
	permit.setConstructionPlan(constructionPlan);
	// no constructionPlan.addPermit()!!!
	session.persist(permit);
	});

view raw ConstructionPlanTest.java hosted with ❤ by GitHub

As me and my colleague say "A seemingly innocent first step on a slippery slope to hell."

Of course, this kind of abuse has always been possible, but it is now easier to do than ever (without having to create an entire class for the priviledge).

References

[1] Wikipedia - Loose coupling

https://en.wikipedia.org/wiki/Loose_coupling

[2] Vlad Mihalcea - The best way to do batch processing with JPA and Hibernate

https://vladmihalcea.com/the-best-way-to-do-batch-processing-with-jpa-and-hibernate/

MERGE SQL Statement

I recently had to do a insert on the database, if the records weren't there, and an update if it was.

Performance was a problem.

Colleague of mine had the same problem when using an insert and update statement with an (exists (select 1)) on 2.5 million records.

After 5 minutes, 5000 rows were done. It would have taken 41 hours to fix them all.

After a merge¹ statement, the entire thing was done in 2.5 minutes.

So, I had to get to grips with the merge statement as well. I hadn't used it before.

	MERGE INTO sourcematerialmetadata smmd
	USING (SELECT sourcematerialid, objectid from sourcematerials, objects
	) info
	ON (smmd.objectid = record.objectid)
	WHEN MATCHED THEN
	UPDATE SET smmd.sourcematerialid = info.sourcematerialid
	WHEN NOT MATCHED THEN
	INSERT (sourcematerialmetadataid,
	objectid,
	sourcematerialid,
	itemid)
	VALUES(seq_s_sourcematerialmetadataid.nextval, info.objectid, info.sourcematerialid, null);

view raw merge.sql hosted with ❤ by GitHub

There's plenty of tutorials on how it works.

Oracle allows a DELETE statement in the MERGE statement, which seems not to be defined in the SQL Standard.

References

[1] Wikipedia - Merge(SQL)

https://en.wikipedia.org/wiki/Merge_(SQL)

[2] Oracle PL/SQL Documentation - MERGE

https://docs.oracle.com/cd/E11882_01/server.112/e41084/statements_9016.htm#SQLRF01606

Proper use of Optional

Using Optional is hard in the beginning.

Especially for first-timers one has a tendency to use the wrong methods.

Take the following sequence of refactoring for example:

First without any kind of Optional:

	public abstract Person findByName(String name);
	public void processInvitations(String name)
	{
	Person person = findByName(name);
	if (person == null)
	{
	return;
	}
	List<Person> friends = person.getFriends();
	for (Person friend : friends)
	{
	sendInvitation(person, friend);
	}
	}

view raw OptionalRefactoring1.java hosted with ❤ by GitHub

A first bad attempt at an optional:

	public abstract Optional<Person> findByName(String name);
	public void processInvitations(String name)
	{
	Person person = findByName(name).orElse(null);
	if (person == null)
	{
	return;
	}
	List<Person> friends = person.getFriends();
	for (Person friend : friends)
	{
	sendInvitation(person, friend);
	}
	}

view raw OptionalRefactoring2.java hosted with ❤ by GitHub

It looks like the person just... kind of removed the Optional back to a situation that he knew and was more familiar with.

Let's try this again:

	public abstract Optional<Person> findByName(String name);
	public void processInvitations(String name)
	{
	Optional<Person> personOptional = findByName(name);
	if (!personOptional.isPresent())
	{
	return;
	}
	Person person = personOptional.get();
	List<Person> friends = person.getFriends();
	for (Person friend : friends)
	{
	sendInvitation(person, friend);
	}
	}

view raw OptionalRefactoring3.java hosted with ❤ by GitHub

Now this is a good try. It is very common to see the two Optional class methods being used together. isPresent() and get().

But we can do better.

It's often the same: we wish to do something, only when there's a value available.

Like so:

	public abstract Optional<Person> findByName(String name);
	public void processInvitations(String name)
	{
	findByName(name).ifPresent(person ->
	{
	List<Person> friends = person.getFriends();
	for (Person friend : friends)
	{
	sendInvitation(person, friend);
	}
	});
	}

view raw OptionalRefactoring4.java hosted with ❤ by GitHub

Of course, it's sometimes better to create a method reference:

	public abstract Optional<Person> findByName(String name);
	public void processInvitations(String name)
	{
	findByName(name).ifPresent(this::processInvitationsOfPerson);
	}
	private void processInvitationsOfPerson(Person person)
	{
	List<Person> friends = person.getFriends();
	for (Person friend : friends)
	{
	sendInvitation(person, friend);
	}
	}

view raw OptionalRefactoring5.java hosted with ❤ by GitHub

But I do find this proliferates the number of methods, but it's an acceptable side effect.

We finally arrive at something concise and easy to read, once you get the hang of it.

Antisamy - false positive security issue

Some time ago, I received a security notification from Github regarding a vulnerability^{1 2} in Antisamy.

It took me a while to find out that it was a false positive.

But the issue in [1] did mention that Antisamy seems to be lacking a maintainer. A worthwhile replacement was suggested as being HTML Sanitizer^{3 5 6}.

I started using the HTML Sanitizer in my project and I like it. The fact that I don't need an XML configuration file is a plus.

There's a StackOverflow answer on why and how of the two projects⁴.

Only thing missing are the policy configuration files from AntiSamy, which are not present in any way in HTML Sanitizer, so you either have to write your own or get them from here⁷.

The file is called HTMLSanitizerAntiSamyPolicy.java.

References

[1] Github - Antisamy issues

https://github.com/nahsra/antisamy/issues/32

[2] CVE details - CVE-2018-1000643

https://www.cvedetails.com/cve/CVE-2018-1000643/

[3] OWASP Java HTML Sanitizer Project

https://www.owasp.org/index.php/OWASP_Java_HTML_Sanitizer_Project

[4] Java: Owasp AntiSamy vs Owasp-java-html-sanitize

https://stackoverflow.com/questions/28577738/java-owasp-antisamy-vs-owasp-java-html-sanitize/29259874

[5] GitHub - OWASP/java-html-sanitizer

https://github.com/OWASP/java-html-sanitizer

[6] MailingList - HTML Sanitizer

https://lists.owasp.org/pipermail/owasp-leaders/2011-March/004883.html

[7] GitHub - ESAPI/esapi-java-legacy - New file to use Java HTML Sanitizer that implements old AntiSamy poli…

https://github.com/ESAPI/esapi-java-legacy/commit/d48e5a6f07601322c44c113058526eb133b777a5#diff-72ca1caf39f169db1ab83d2af2ec6cc3l