Showing posts with label courses. Show all posts
Showing posts with label courses. Show all posts

Thursday, 21 December 2017

CTF training

On the 14th of December, our company decided to hold a CTF training provided by Dennis van Halteren and Joep van Gassel of Nováccent Group1 for us poor software developers.

I had never heard of it before, so I was kind of interested.

Capture the Flag is a competition where different teams compete to each find the most "Flags": issues in especially for the competition buildt pieces of software. Different flags have different scores. The Application you are trying to hack will let you know in different fashions about the flags you have found. Some flags are hard, some are easy, their respective points reflect this.

The CTF training consisted of an introduction into security and how to test it, and the main CTF competition.

One of the good points of the introduction, was regarding some of the tools that are used in security and penetration testing, which I will mention here:

BeEF3
a tool for injecting bad stuff into browsers of users by means of a vulnerable website
Kali Linux4
a Linux OS containing a lot of penetration test tools and security tools by default

The introduction was very general and lacked depth and lasted about an hour. I already learned most of it during a session working with WebGoat from OWASP2 some years previously.

The CTF competition was a lot of fun. We sorted ourselved into groups of two or three software designers. Each group was provided with a key with which to enter the portal from which all the assignments could be started. A dashboard showing the different scores was available on the big screen.

A number of web portals were provided to us, which all contained different challenges for us to overcome:

Some of the security issues to detect were:

  • caching
  • escalating priviledges
  • using left-over files that should have been deleted
  • SQL injection
  • using error messages to gain insight into the backend
  • wrong permissions that allow too much
  • examining web code for vulnerabilities (HTML, JavaScript, CSS)

The intensity was very high, all of us competing in finding the most flags the fastest. The combination of thinking very hard, trying stuff, frustration if it didnt't work, the cooperation and communication between team members, made most of us feel quite exhausted at the end of the day.

There was no followup after the competition, which I felt was a pity. The trainers were available for questions regarding some of the assignments during and after the competition.

References

[1] Nováccent Group
https://www.novaccent.nl/
[2] OWASP WebGoat
https://www.owasp.org/index.php/Category:OWASP_WebGoat_Project
[3] Kali Linux - a free and open-source penetration testing platform
https://www.kali.org/
[4] BeEF - The Browser Exploitation Framework
http://beefproject.com/
Practice CTF List / Permanant CTF List
http://captf.com/practice-ctf/
Wikipedia - Wargame (hacking)
https://en.wikipedia.org/wiki/Wargame_(hacking)

Thursday, 17 November 2016

Workshop Continuous Delivery with Docker

Last Monday, on the 14th of November, I got a chance to attend a little workshop on Docker organised by OV Software. It was at the NH Hotel near the facilities of OV Software in The Hague. It is a little far from my place of residence, but I could tag along with a colleague of mine, which helped.

They did a great job of preparing the workshop properly. A VirtualBox image was provided (https://goo.gl/XUM898). The downloaded file can be "imported" (File -> Import Appliance) into the VirtualBox platform (version minimum is 5.0).

Unfortunately, my labtop is not one of the greatest, and the amount of memory required is too much for it. I had to borrow a labtop from work. If this keeps up, I'll have to buy something better for myself one of these days.

Also, unfortunately, I was unable to attend a previous workshop of which this workshop is a followup, so I did miss some background on Docker, and this was my first foray into this new area. Thankfully, a lot can be found on the internet1 2, and I already had plenty of experience with Linux, Java, Maven, and just software development in general.

I had small problems getting it to run, for example:
  • I had Vx extentions turned off in the BIOS of my labtop. I needed to turn them on.
  • PAE - Physical Address Extentions was turned off. It should be on.
  • Somehow, when importing the image, I had turned the machine to 32bits. It's a Linux 64bits image.
  • It seems it sometimes helps to turn off your USB in the Virtual Machine settings.

The workshop was provided by an employee of OV Software, Jeroen Peeters, who is an excellent and calm speaker.

One of the first things where I went wrong is thinking that Docker is a kind of Virtualisation. It is not. Docker runs within the same Operating System and the containers also all run in the same Operating System. The images, however, can contain everything the container needs to properly run the image. So, they are simply processes running under the same kernel with the same kernel libraries at their disposal, but it is true that these processes are a little better isolated from the rest of the system.

There are plenty of alternatives to Docker, but Docker seems to be the main menu these days:
  • LXC
  • Rocket, tooling for developers and systemmanagement
  • Lmctfy
  • OpenVZ
  • Canonical LXD, a Ubuntu thing but more isoltation, so more Virtualisation as it also runs system images

To be more specific, the workshop was regarding Continuous Delivery3 4 with Docker.

During the workshop we set up the following Docker containers inside Docker running on Linux Lite 3.0 LTS (Ubuntu based):
  • a container of GitLab that held a git repository of a Java Program
  • a container of Jenkins
  • a maven container (for inside Jenkins)
  • a tomcat container
  • a load balancer container (Haproxy)
  • a testx container6

So a webhook in the GitLab repo made sure that jenkins was notified if the source code had changed. Jenkins would therefore start a build pipeline, that would build the application and deploy it on Tomcat and would start testing it using TestX5 (a frontend tester tool sauce over protractor and Selenium).

The github repository containing the Java application can be found at https://github.com/jeroenpeeters/docker-workshop-java-todolist/. It is a simple TODO list registration tool.

All this is running in one virtual server. For clustering and the like, you could have a look at Kubernetes, the google solution for running Docker on several machines. Complexity is quite high. But there are also other solutions.

It looks very interesting if you are interested in using Continuous Delivery. I do know that System Management (where I work) is very apprehensive about it. I think they fear to lose control on what Developers can do. We are already working with it in some fashion, but it needs to be improved.

References

[1] Docker
https://www.docker.com/
[2] Docker - Installation on Fedora
https://docs.docker.com/engine/installation/linux/fedora/
[3] Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
https://www.amazon.com/dp/0321601912?tag=contindelive-20
[4] MartinFowler.com - ContinuousDelivery
http://martinfowler.com/bliki/ContinuousDelivery.html
[5] TestX
https://github.com/testxio/testx
[6] TestX Docker Image
https://hub.docker.com/r/testx/protractor/
Nginx - Docker for Java Developers
https://www.nginx.com/resources/library/docker-for-java-developers/
The Docker Book - Containerization is the new virtualization
https://www.dockerbook.com/
Jenkins
http://www.jenkins.io

Thursday, 29 September 2016

Specification By Example

“The formulation of the problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill.”
- Albert Einstein

The image on the left is a Dutch proverb, meaning "Setting a good example to follow". It exemplifies the Specification by Example idea.

In short, I managed to follow a one-day course on Specification By Example1, by InfoSupport2 at my place of work (which was convenient) along with a collection of my colleagues.

It seems that in the past there were different terms for the same thing, for example ATDD (Acceptance Test Driven Development) and BTDD (Behaviour Test Driven Development). Martin Fowler shared all these under the heading of Specification by Example.

The main goal here is to create living documentation, documentation that evolves along with the software and is therefore always up to date. It also provides a common domain language that can be used by everyone in the company, be it business analysts, software developers, project managers, stakeholders, product owners, etc.

Once proper examples are created using this domain language, these examples can function as the templates for the creation of tests. At work we use Gherkin and Cucumber4 to get these examples written down.

Along with the course I was provided with the book3 on Specification By Example, and I look forward to reading it.

References

[1] Trainingen - Driving Development by Example - InfoSupport
https://training.infosupport.com/trainingen/driving-development-by-example
[2] InfoSupport
http://www.infosupport.com/
[3] Specification by Example - How Successful Teams Deliver the Right Software
Gojko Adzic
[4] Wikipedia - Cucumber (software)
https://en.wikipedia.org/wiki/Cucumber_(software)
Impact Mapping
https://www.impactmapping.org/
Jeff Patton, Product Manager, Agile, Lean, UX and Product Design Evangelist
http://jpattonassociates.com/

Thursday, 22 September 2016

Final project for DEV201x Introduction to TypeScript on edX.org



I have just now completed the course on typescript1 available at edX2.

The good part about the course is that it is a decent introduction into TypeScript, and also provides information on why typescript was invented in the first place. However, that is basically all it is. To become adept at TypeScript, besides knowing TypeScript, it is essential to have a pretty good grasp of JavaScript.

It was a good course, and helped me a lot, but it is a small course indeed.

edX vs. Coursera

This is also the first time that I got to try edX2 as I previously only had experience with Coursera3. I don't think it is fair to compare the two, as it very much depends on which course you take. I do think the web interface of edX, in total, could use a little more polishing up, as some of the user interface quirks were hard to understand.

Coursera, I think, is a bit more polished and provides a better experience.

Final Project

The Final Project consisted of a small web application that displays information regarding Painters. I basically did the bare minimum, as I find it hard to find the time to do anything constructive after working hours. (What with life and everything).

What helps is the little IDE provided for free by Microsoft called Visual Studio Code4.

The animated gif below shows the user interface of the little web application written in TypeScript5.

Click on it for a bigger and better view.

References

1. Microsoft: DEV201x Introduction to TypeScript
https://courses.edx.org/courses/course-v1:Microsoft+DEV201x+1T2016/info
2. edX | Free online courses from the world's best universities
https://courses.edx.org/
3. Coursera : Online Courses From Top Universities
https://www.coursera.org/
4. Microsoft - Visual Studio Code
http://code.visualstudio.com/
5. Typescript Language
http://www.typescriptlang.org/

Thursday, 12 May 2016

Typescript

Our architect recently let us know that there is an Introduction to TypeScript1 available on the edX Website2.

Naturally, given that the TypeScript I am using at work is still very hard for me to understand, and given the fact that I wish to start using Angular 2 (which uses Typescript extensively) at home, I found this to be fascinating.

It was only a matter of minutes for me to enroll.

The course starts on May the 16th and is given by Microsoft experts and apparently the inventor of the TypeScript language.

I am quite looking forward to it and will let you know my thoughts at the end of the course.

References

[1] edX - Introduction to TypeScript
https://www.edx.org/course/introduction-typescript-microsoft-dev201x-1
[2] edX - Free online courses from the world's best universities
https://www.edx.org/
Microsoft Virtual Academy
https://mva.microsoft.com/
Typescript Tutorial - Quick Start
http://www.typescriptlang.org/Tutorial

Sunday, 19 April 2015

Java 8 - Beta Exam Attempt

Well, last Wednesday, 15 April 2015, I tried to get certified in Java 8 by taking the Upgrade to Java SE 8 Programmer Exam. It contained 126 questions and I was given 3 hours to provide all the answers.

For more information on what topics are required reading, see reference [1].

Given the fact that I am not yet comfortable with the new features of Java 8, answering the questions took a lot of time. I was unable to complete the last 12 questions due to this.

The beta exam is now no longer available.

I'll have to wait approximately 11 weeks for a result, though. A long time to wait, but that's always the case with Beta exams.

At least it's behind me now. Next up is trying to get certified with Scrum.

Thoughts

Here are some of my thoughts:
  • there were a number of 'trick' questions
  • learn the method signatures of java.util.function, luckily the naming scheme is extremely consistent
  • learn the difference between the source of stream, the intermediate operations, and the terminal operations. Here's some pointers:
    • a stream, once evaluated, can no longer be used. Trying causes an InvalidStateException.
    • omitting a terminal operation, causes the Stream to not be evaluated at all.
    • using two terminal operations, of course, causes a compile error
    • a stream doesn't change the underlying source
  • learn the additional operations now available (using Lambdas) on the already existing Collections of the Collection framework
  • learn when an interface is a functional interface. Here's some pointers:
    • @FunctionalInterface is not a requirement
    • what happens when you add default methods?
    • what happens when you add an abstract method?
    • what happens when you add an abstract method, whose signature is equivalent to an existing method in the Object class?
    • what happens when you add a static method?
    • what happens to a functional interface that inherits from another interface
  • Remember what Daylight Saving Time is, the clock is advanced by one hour during summer months (in March), the clock is returned to normal time (turned back one hour) during the winter months (in October)
  • practice,... a lot ([2] is a good reference with exercises)

References

[1] 1Z1-810 Upgrade to Java SE 8 Programmer
https://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=5001&get_params=p_exam_id:1Z0-810
[2] Java SE8 for the Really Impatient
Cay S. Horstmann

Sunday, 1 February 2015

Java SE 8 Certification


SInce the beginning of the year, this year, 2015, it is possible to enter the Beta Exam for the Oracle Certified Professional, Java SE 8 Programmer.

Naturally, I'm going to give it a go.

The Beta Exam has only recently been (informally) announced, so it might be a while before I can post any more information about how it was.

References

[1] 1Z1-810 Upgrade to Java SE 8 Programmer
https://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=5001&get_params=p_exam_id:1Z0-810

Saturday, 19 July 2014

Successfully completed Functional Programming Principles in Scala

I earned 97.7% with distinction.

Evaluation

One can tell that the subject matter was Academic, with a firm grounding in Mathematics, which appealed to me.

The assignments provided a lot of information on what is expected, so there are no surprises, but you do need to read carefully.

The one assigment that provided the most difficulty was assignment 6, regarding the discovery of Anagrams of a sentence.

I had to wrestle a bit with the Scala syntax. It's new for me.

I especially found foldLeft and foldRight counter-intuitive sometimes.

I learned a lot on the following topics, in no specific ordering.
Scala Programming Language
by the creator, Martin Odersky, himself.
Functional Programming
one of the main subjects of the course
Domain Specific Language
Scala provides several ways to program according to a domain model2, instead of a technical/software model
Mathematics - Set Theory
the code is very close to the mathematical theory. Purposefully crafted that way, of course. It means we can actually use mathematical operators (some of the time).
Behaviour Driven Development
you can write tests that read more naturally
Test Driven Development
assignments had to pass certain tests (that are unknown), so your own tests had better be complete/sufficient
Recursion
we used a lot of recursion, you do not see that in "normal" programming languages.
It was a huge amount of fun to do, both to learn a new Programming Language1 and to learn a new Programming Paradigm.

References

[1] Coursera - Functional Programming Principles in Scala, by Martin Odersky
https://class.coursera.org/progfun-004
[2] Wikipedia - Domain Model
http://en.wikipedia.org/wiki/Domain_model

Thursday, 24 April 2014

Coursera - Functional Programming Principles in Scala

On the 25th of April, meaning this Friday, I start my course on Scala.

I found Coursera1, an education platform that partners with universities to provide courses online for free.

I am really excited in finding out how Scala differs from Java, and what new programming methods I'll learn.

The fact that the Course2 has been set up by the creator of the language, Martin Odersky of École Polytechnique Fédérale de Lausanne, is a big plus.

I hope I can meet the deadlines.

References

[1] Coursera
https://www.coursera.org/
[2] Coursera - Functional Programming Principles in Scala
https://www.coursera.org/course/progfun

Wednesday, 29 January 2014

1Z0-807 Java EE 6 Enterprise Architect Certified Master Exam

follow-up of [2]

Pfew! Made it! Woohoo! I had answered 73% of questions correctly, and I needed 71%. I should've studied more.

It is Multiple-Choice, and some of the wrong possible answers are really quite close to correct.

Then again, who has to answer a Architectural Question in their business careers in less than 2.5 minutes on average, really?

On to the next step, the Assignment!!

The exam

Most of the exam questions were regarding business use cases.

The exam questions can be broadly separated into three distinct categories:
use cases
of imaginary companies seeking to change/re-evaluate (a part of) their current outdated infrastructure in some fashion
questions on design patterns
when to use them, what their characteristics are, etc. *)
questions on JEE technologies
when would you apply which technologies, given some Non-Functional Requirements (NFR)

*) Seems the books "Core J2EE Patterns", "Real World Java EE Patterns: Rethinking Best Practices" and "Design Patterns: Elements of Reusable Object-Oriented Software" really did help, and were fascinating reads in and of themselves.

References

[1] 1Z0-807 Java EE 6 Enterprise Architect Certified Master Exam
http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=5001&get_params=p_exam_id:1Z0-807
[2] Architect Enterprise Applications with Java EE - Course
http://randomthoughtsonjavaprogramming.blogspot.nl/2013/09/architect-enterprise-applications-with.html

Wednesday, 25 September 2013

Architect Enterprise Applications with Java EE - Course

I had the opportunity to attend the Oracle course on "Architect Enterprise Applications with Java EE"[1] from Monday 09/09/2013 to Friday 13/09/2013. (after forking over a big sack of money)

The Course does have a number of Lab exercises, mostly based on UML or open questions (that created several discussions, which I enjoyed immensely). UML diagrams were drawn using UMLet[3].

I took the course as I wish to prepare for the 'Oracle Certified Master, Java EE 6 Enterprise Architect' certification.

This is the first step in the Certification Process [2].

I did receive some very interesting Course material along with it. However, it's been licensed to me specifically. (I could let you read it, but then I would have to kill you. Oracle licensing rules say so.)

References

[1] Architect Enterprise Applications with Java EE
http://education.oracle.com/pls/web_prod-plq-dad/ou_product_category.getAllProductsPage
[2] Oracle Certified Master, Java EE 6 Enterprise Architect
http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=458&get_params=p_track_id:OCMJEE6EA
[3] UMLet
http://www.umlet.com/

Wednesday, 5 December 2012

Continuous Delivery


On the 30th of November 2012, I was able to attend a training on Continuous Delivery, courtesy of Xebia. I received the offer, as a member of the NLJUG.

It is the first training I've witnessed where they provide you with an entire OS image, for easy starting of the training. I guess they're professionals who have worked with this before. They're using VirtualBox for this. Of course, it does mean you have to download a 1.3 Gb image beforehand.

The different technologies contain:
  • Subversion as version control system
  • Java for the programming (although no programming was actually done during the workshop)
  • maven for building
  • Jenkins for executing automated build- and packaging activities
  • Fitnesse for automated testing
  • DeployIT for automated deploying
  • Apache as the lowlevel webserver
  • JBoss as JEE application server
  • MySQL as the database
A note of criticism. The workshop mail mentioned working with Selenium, but it was only mentioned once, and briefly at that, during the entire day.

VirtualBox


I was much impressed with VirtualBox. It enabled me to start without having to configure a lot of software on my native OS (Fedora 16 86_64).


Bear in mind, you need the kernel headers installed. Also bear in mind, something which had me stumped at first, that the latest kernel headers are not necessarily of the kernel you are running at the moment. Upgrade your kernel as well, and reboot.

That fixed it for me.

So in no time I was running this Ubuntu precise 32 on this virtual machine on my Fedora core 16.

The Workshop


The workshop started off with a presentation by Mark van Holsteijn, principal consultant of Xebia, regarding what we were actually going to do and what the advantages were of doing it this way.

A lot of organisations create their applications by hand. In a lot of cases it involves an integration/systems engineer following a thick document containing a lot of steps to be performed in sequence. This has the following disadvantages:
  • it is time consuming
  • complexity is high
  • it's error prone, things get forgotten, things go wrong (especially in those cases where an integration engineer is stuck with a "non standard" environment/customer/hardware/etc)
  • it costs manpower, and manpower, contrary to computing power, is expensive
  • time between releases is long
  • a lot of new functionality is incorporated in a new release, increasing the number of possible bugs found in production
  • time to market is long
Continuous Delivery is all about bringing your Java JEE applications fast, flawless and completely automated to production. So from development, to junit testing, functional testing, systems/integration testing, to staging, to production. This is called a end-to-end Continuous Delivery Deployment pipeline.

It fits the Agile way of working. Agile is to satisfy customer through early and continuous delivery of valuable software.

So basically every checkin into svn is a release. It turns out that big releases of functionality are always more bugridden than a lot of releases with small added functionality. It is an example of the old mantra "release early- release often".

A quick incomplete first attempt at a new feature, will also provide valuable info on if your customers actually want this functionality. You can tell, by checking if the new functionality is actually used.

Jenkins

At the end of the workshop, we ended up with the pipeline in the picture below.
The workshop explained briefly all the following plugins in Jenkins we were going to use. They can be downloaded from jenkins-plugin-hub.heroku.com.
Parameterized Trigger
This was used to trigger another job from the current job, and to provide information for this next job, for example revision numbers. Essential if you wish to use the same build between jobs. In our workshop, the parameter PL_SVN_REVISION was provided by the first job as one of the post build actions, and passed onto the next jobs.
In the next jobs, checkbox "This build is parameterized" was checked and the parameter to be imported entered. the parameter could be used as ${PL_SVN_REVISION}. In our case it was frequently used for:
  • source code version management, making sure the proper revision was checked out in svn, using "http://10.20.20.20/svn/sample-app/cd-fitnesse-runner@${PL_SVN_REVISION}",
  • in the build parameters, " -Drevision=${PL_SVN_REVISION}" and
  • in the name of the build
Rebuilder
A simple plugin that rebuilds a parameterised build, by means of a button in the options of the build in question.
Maven Repository Server
changes Jenkins into a Maven Repository Server. The repository is defined in the settings.xml file of your jenkins installation, in our case it was appropriately called "jenkins". You can tell maven to use your profile defined in the settings.xml file using "-Pjenkins" as a parameter. The downstream jobs can define a Upstream Maven Repository in the Build Environment. We have set it to "../everything" to get all build artifacts from previous jobs.
Build Pipeline
provides a view of upstream and downstream connected jobs that typically form a build pipeline. A screenshot is provided above.
Environment Injector
useful for configuring your buildenvironment per job
Throttle Concurrent Builds
so we cannot build more than a specific number of builds in parallel. We haven't configured it, as it is not really an issue in our current setup.
Priority Sorter
allows for the build queue to be sorted based on pre-assigned priorities for each job. For example a smoke test has high prio.
Promoted Builds
A good way to distinguish good builds from bad builds, for example in such cases that a single job is not indicative of the health of the overall build. A build could get promoted if the jobs of junit testing, integration testing, and staging completed successfully.
Build Name Setter
An excellent little plugin that allows descriptive names for builds instead of the default jenkins build "#1", build "#2" etc. It can be defined in your build environment of your job. In our case it was defined as "#${BUILD_NUMBER} - rev ${ENV,var="SVN_REVISION"}", so we got "#1 - rev 34" as name.
Deployit
creates and uploads deployment packages (dar, deployment archive, yet another abbreviation of a specific jar) using artifacts in the jenkins workspace. Very important post build action in our deploy job.
Wall Display
for displaying on your bigscreen on the wall, to see what builds are running in what state, etc. Management will love it.
The first job, that triggers subsequent jobs, is set to "Poll SCM" about every minute (cron syntax * * * * *) to check for changes in SVN. So any changes are automatically picked up and also trigger a build.

FitNesse

FitNesse is:
  • software development collaboration tool
  • software testing tool
  • wiki
  • webserver
It is possible to run FitNesse standalone, it will start a small webserver, like this.

But during the workshop we also integrated it into Jenkins.

DeployIT

DeployIT is an application developed by Xebia Labs. A fairly big amount of time was spent on it during the workshop, but I don't blame them. They have to make a living too.

It automatically can deploy an application to multiple application servers, using a set of standard scripts (that can be modified if desired). It is executable from Jenkins, through a plugin they developed.

The advantages are that it is a general component and that it has knowledge of most application servers and how to deploy applications to them. It also can determine what needs to be deployed, has a lot changes or has little changed, deploy a little or deploy everything. It also contains monitoring.

You will have to create a DAR (Deployment Archive) package, that can be used by DeployIT to deploy to different servers. Advantage is that the package is environment independent.

DeployIT uses OverThere, an Open Source Framework library which knows how to connect to different operating systems, and to execute operations once there.

One of the tabs, "Repository" can show you the "world image" of DeployIT. There are configured which application server is where, etc.

It is possible to add Sql Scripts to the Deployment Package, but Sql Scripts, as usual, are a different thing altogether, because it deals with 'state'.

DeployIT can rollback applications to previous versions automatically if the deployment failed, but this is of course not automatically possible with Sql Scripts. It is possible to add special rollback scripts that are executed on the database upon a rollback to a previous version of the application.

DeployIT has security inside to determine if someone, once they are logged on, has the appropriate rights to deploy somewhere.

One of the features I really really like about DeployIT, is that it provides a central place where is administrated where all the revisions are installed, on which Application Servers they are deployed. No more doubt about what is installed where.

Tools


Some tools were not addressed in any way, but I managed to make notes of them anyways:
GreenPepper
is a tool integrating executable specifications and automated functional testing into software development processes, thus reducing any ambiguity related to the expression of needs between all actors.
Selenium
automates browsers
jmeter
load test functional behavior and measure performance

Requirements

There are some requirements in performing builds this way.

Your automated tests have to be very good, very thorough and fairly complete. Otherwise the whole process breaks down.

You and your team have to organise yourself around products and align on common goals. It's very important that the systems and integration teams are onboard with this.

Sunday, 4 November 2012

J-Fall 2012 Report

J-Fall 2012[1] has come and gone, and I write a small blurb on what I managed to learn at each session.

It was a great J-Fall. I had a lot of fun and learned a lot from listening to some of the people, who were so passionate about the particular field they worked in. I hope next time it will be equally interesting. Or even better! Keep it up, y'all!!!

It's always interesting to see any big event that has a line for the mens room, instead of the ladies room.

Your Product Owner is Just Better at Pretending

Speaker(s): Erwin van der Koogh

It was extremely early in the morning, and I had to catch the very first train to the Conference, or I would miss out.

Some statistics. Of all the features of a software product, only 40% of those are actually used by the customer.

In which case it is very important to decide what to include in the product and what not to. Also, it might be obvious, who your customer is, but sometimes things are not as cut and dry.

The powerful example of Facebook was discussed. The Customers of Facebook are not the users, but the advertisers. Advertising is how Facebook makes (hopefully enough) money. The users are the Product. They are the reason Facebook can charge for advertising. This will make you look at Facebook in a whole other light.

Another statistic is that, if you were to eliminate your source code, and you'd have to write it again, but with the knowledge and experience you have now, how long would that take? Just as long? Twice as long? Half as long? It turns out around 1/3 to 1/4 of the time spend on a project is spent coding. Yet that is the only part that is currently being considered for efficiency in all our Software Development Methods we have gotten used to.

A third I wish to point out, was Documentation. Nobody (seems to) reads documentation. I always say that I prefer to have bad documentation instead of no documentation. I always have the optimistic view that if there is bad documentation, the incentive to fix it is greater than if there is no documentation to be the first to write it.

The speaker did not share my opinion. He mentions that bad documentation, if there is no hint that it is in fact bad, could be assumed to be correct and thusly provide misplaced confidence in what you are doing, leading to larger problems later on.

One other instance is to show people that there's a new Upgrade/Component/Widget/whatever available on your website that does X for you. Then place a link under it to a 404 page. Then run statistics on how often that 404 page was accessed. It gives you an idea of how much a feature is wanted, without writing any actual code. A lot of these ideas were expressed here.

I was sufficiently intrigued that I think there's more than one blogpost in here.

Keynote - Oracle


Stephen Chin @steveonjava Java Technology Ambassador and JavaOne Content Chair was late for the Keynote. He was on a Nighthacking tour on his bike through Europe attending all the major conferences.

But he did manage to get there, riding into the conference room, on said bike and biking gear.

They made quite a show of it.

Find more complete info on Geertjan's Blog.

I really liked the picture of 20 Raspberry Pies in a 8U unit.

Java EE 7 Platform Overview and Highlights

Speaker(s): David Delabassee


They are hard at work for the new version building new APIs and improving old ones.

new API:
  • JSONP presentation view
  • JSON API for Java
  • java.net.websocket
  • batchapplication for Java 1.0
  • java temp caching -> distributed across nodes?

old api to be updated:
  • rest + hypermedia + client api
  • JMS
  • Bean Validation
  • JSF 2.2
  • github.com/jersey./hol-sse-websocket

The Aquarium

javaee-spec.java.net

Some of the Specifications are not quite finished yet. So if you have something to say about how things should be implemented/work, you still have a chance to mention it.

Java EE Multi-tenancy in Practice

Speaker(s): Frans van Buul

Good stuff. Multi-tenancy here is having multiple customers in the same database, and your application is minimally impacted.

It is concerning adding a column to the tables (which can automatically be done by Oracle and by PostgresSQL Enterprise) that indicates the Tenant.

The idea is to have just one database, where every table can be "viewed" by tenants. The tenants will only see the information (rows) of himself and not of other tenants. This way, the impact in the code is minimal.

In a worse case scenario, we have to do all this stuff ourselves. This means that the code will be littered with a lot of if statements to double check if the proper tenant is inserted etc.etc...

In hibernate it is possible to "fix" the problem (though in Hibernate 5.0 there might be a solution for the problem) with an Interceptor (or a PrePersist) to fix the tenant properly on the entity by Lookingup in the context for the proper tenant. A good way of retrieving the tenant is by using the hostname for the proper information. For example holidayinn.localhost or novotel.localhost.

What I find most admiring is the fact that he did make some mistakes, and he managed to find out where he made those mistakes, and fix those mistakes.
Mistakes made were:
- tomcat selected instead of glassfish, and tomcat has a web profile, which isn't everything he needs

MySQL workbench was awesomeness! He generated the entire database from scratch every time, by executing a script created by MySQL workbench!

Combination Netbeans and Glassfish and MySQL and EclipseLink.

Netbeans never ceases to amaze me, the quick way in which an entire project can be started and generated in a manner of minutes.

EclipseLink supported a special annotation to get the Tenant system working,

One question in the public was that there were small and big companies and that the big companies, with their many permutations, would push the little companies out of the cache.

In the case of Frans, his organisations were all on the same scale and he didn't see this happening in his practice.

My opinion: the little companies do not need to be in the cache, as their involvement isn't that big to begin with. They can wait for their data a bit longer.

In general a very tricky question is the second level cache. It is possible to do all this by hand in MySQL, but you basically "screw" with the Primary keys. The primary key is partially "hidden" from the application, the tenant part to be precise. So the primary key known to the system might be the primary key of another record as well (but with a different tenant). This is a problem with caching, as the cache might provide the wrong instance. It is best in those case to just turn the second level cache off.

Question of the audience was: just make the primary key a autogenerated id, problem solved. Frans agreed in principle, but what if you have a customer that wishes to port his local database into your tenanted database. You'd have to resequence all the primary keys before inserting into your database. This is a complicated and error prone process. Not to mention that this must be done on a production database, during a time of low load, so basicallly at night.

Microsoft Keynote


Microsoft explained the advances made with their Windows Azure, their cloud solution. Apparently it is no longer Windows centered, but can support a number of different technologies. Even Linux was mentioned!

Also deploying to a staging environment, testing and then switching the staging and the production environment at the load balancer level to go live within a second was nice.

Hands-on Lab: RRRADDD! ... Really Ridiculously Rapid Application Development (Domain-Driven)

Speaker(s): Dan Haywood, Jeroen van der Wal

It concerns an Apache Incubation project called Apache Isis. It is still in its infancy but looks nice.

It reminds me the most of something we're building at work. It works on annotations for displaying your data on webpages without much hassle.

Hands-on Lab: MongoDB

Speaker(s): Maikel Alderhout

Database evolution in short:
  • 1990: Oracle RDBMS
  • 2000: RDBMS and OLAP/BI
  • 2010: NoSQL Hadoop

Several trends have a big impact on the database landscape:
  • data volume, type & use
  • agile development
  • new hardware architectures, cloud, commodity

4 categories of NoSQL are available:
  • key-value stores
  • document based
  • columnfamily/bigtable clones
  • graph databases

MongoDb is a scalable, high-performance NoSQL databaae of the document based type.
  • no transactions
  • json documents binary stored
  • replication/sharding/durability.

No strict data schema. Examples are
Twitter and Foursquares. Foursquares is actually running on MongoDB.

j:true => this is the system used by relational databases. Changes are stored in a journal on disk. MongoDB can be set to what you want:
  • asyn(default) instant feedback, "got the message I'll get around to it"
  • w=1, "ĂŹ'll remember it"
  • j:true, "Wrote down what I need to do"
  • w=majority, "Wrote everything down"
  • w="<tag>", "Wrote everything down multiple times"

The command line client during the lab felt like working somewhere between SQL and calling javascript functions.

The situation:
  • RDBMS -> a lot of functionality, very little flexibility
  • memcache -> little functionality, a lot of flexibility
MongoDB for the most part goes a long way towards RDBMS feel.

Scala Through the Eyes of Java (8)

Speaker(s): Urs Peter

Urs Peter is a Speaker/Trainer of Xebia and provides courses in Scala, and it shows.

Scala started out in 2003. Created under the EPFL by Martin Odersky and exploited by the company Typesafe.

Has the following frameworks, Scala + Akka + Play.

Some points that came up:
  • syntax lightweight (helps?)
  • val = final
  • var = field
  • none = ?
  • operators are just methods (except == and a few others)
  • types are inferred.
  • no more NullPointerExceptions
  • functional programming (first class citizens)
  • object oriented programming (all the way, no native types)
  • multiple inheritance by means of traits.

An example was given using spaceships, always cool.
  • base, has the following traits
    • shield
    • gun
    • medic
  • commander, has the following traits
    • shield
    • gun
  • fighter, has the following trait
    • gun
  • mechanic, has the following traits
    • shield
    • medic

Dutch Scala Enthousiasts

Scala for the Impatient.

Shadaj Laddad

References

[1] J-Fall 2012
http://www.nljug.org/jfall/

Wednesday, 24 October 2012

J-Fall 2012

The NLJUG[1] is once again organising J-Fall 2012[2]. It takes place on the 31st of October 2012 in Nijkerk, in the Dutch province of Gelderland.

I shall be visiting, and see what new things I can learn.

I hope to write some blogs about it. As such I thought I'd post my current programme here.
Time: 08:00 - 08:50 Early Bird sessions
Title: Your Product Owner is Just Better at Pretending
Speaker(s): Erwin van der Koogh

Time: 09:20 - 10:10 General Session
Title: Keynote - Oracle

Time: 10:40 - 11:30 Parallelsessions
Title: Java EE 7 Platform Overview and Highlights
Speaker(s): David Delabassee

Time: 11:35 - 12:25 Parallelsessions
Title: Java EE Multi-tenancy in Practice
Speaker(s): Frans van Buul

Time: 13.35 - 14.20 General Session
Title: Keynote

Time: 14:25 - 15:15 Parallelsessions
Title: Hands-on Lab: RRRADDD! ... Really Ridiculously Rapid Application Development (Domain-Driven)
Speaker(s): Dan Haywood, Jeroen van der Wal

Time: 15:45 - 16:35 Parallelsessions
Title: Hands-on Lab: MongoDB
Speaker(s): Maikel Alderhout

Time: 16:40 - 17:30 Parallelsessions
Title: Scala Through the Eyes of Java (8)
Speaker(s): Urs Peter

References

[1] NLJUG - Nederlandse Java Users Group
http://www.nljug.org
[2] J-Fall 2012
http://www.nljug.org/jfall/