Visualization of the Message Flow between Business Functions with Vaadin and Neo4j

The Project

Analyzing and visualizing the message flow between business functions was the goal of my current project. At first, we considered using a UML tool for this job, but we came to the conclusion that it might not be as flexible as we need it to be. Finally, I’ve got the assignment to create a custom web application.

Since business functions and messages are related to each other, it made sense to represent them as a graph. That’s why I chose Neo4j as the database. Now the question was how to manage and visualize the graph. As I’m experienced with the Vaadin framework I want to use it also in this project. 

Vaadin has a lot of great UI components but in my case, there was no match. Finally, I’ve found vis.js. The network diagram seemed appropriate for the visualization. Luckily Vaadin provides the Vaadin Directory, a place to publish 3rd party components. From the Vaadin directory, a component called vis-network-vaadin is available that provides a Java API on top of vis.js

The Graph

The graph below is a simplified model of what my client wants to manged in the application. A business function can send many messages and a message can be received by many business functions.

The Implementation

First I created a Vaadin project on start.vaadin.com and added the vis-network-vaadin dependency for the visualization. As Vaadin uses Spring Boot by default I could just add spring-boot-starter-data-neo4j for the data access.

Data Access

Spring Data Neo4j provides easy access to Neo4j. As I already know Spring Data JPA and the programming model are very similar it was easy to get started. First I’ve mapped the nodes and defined the relationships using the Neo4j annotations.

@Node
public class BusinessFunction {

    @Id
    @GeneratedValue
    private Long id;

    private String nameDE;
    private String actorsDE;
    private String descriptionDE;
}
@Node
public class Message {

    @Id
    @GeneratedValue
    private Long id;

    private String nameDE;
    private String descriptionDE;

    @Relationship(type = "SENDS", direction = Relationship.Direction.INCOMING)
    private Set<BusinessFunction> senders = new HashSet<>();

    @Relationship(type = "RECEIVES")
    private Set<BusinessFunction> receivers = new HashSet<>();
}

To read and write the data you can use repositories and make use of interface methods that will be used to generate the queries for you. Remark: I didn’t care about the performance so the generated queries were good enough in the first phase.

public interface BusinessFunctionRepository extends Neo4jRepository<BusinessFunction, Long> {

    Optional<BusinessFunction> findByNameDE(String name);

    List<BusinessFunction> findAllByNameDELike(String name, Pageable pageable);
}
public interface MessageRepository extends Neo4jRepository<Message, Long> {

    Optional<Message> findByNameDE(String name);

    List<Message> findAllByNameDELike(String name, Pageable pageable);
}

Diagram

Finally, I had to visualize the graph with a network diagram. Using the vis-network-vaadin API made it quite simple. I just had to map BusinessFunction and Message to nodes and create edges from the relationships.

var networkDiagram = new NetworkDiagram(Options.builder().build());
networkDiagram.setSizeFull();

var businessFunctionNodes = businessFunctionRepository.findAll().stream()
        .map(businessFunction -&gt; createNode("b-", businessFunction.getId(), businessFunction.getNameDE(), "DodgerBlue"))
        .toList();
var nodes = new ArrayList<>(businessFunctionNodes);

var messages = messageRepository.findAll();
var messageNodes = messages.stream()
        .map(message -&gt; createNode("m-", message.getId(), message.getNameDE(), "Orange"))
        .toList();

nodes.addAll(messageNodes);

var dataProvider = new ListDataProvider<>(nodes);
networkDiagram.setNodesDataProvider(dataProvider);

var edges = new ArrayList<Edge>();
for (Message message : messages) {
    for (BusinessFunction sender : message.getSenders()) {
        edges.add(createEdge("b-", sender.getId(), "m-" + message.getId().toString(), getTranslation("sends")));
    }
    for (BusinessFunction receiver : message.getReceivers()) {
        edges.add(createEdge("m-", message.getId(), "b-" + receiver.getId(), getTranslation("receives")));
    }
}

networkDiagram.setEdges(edges);

These are the helper methods to create nodes and edges:

private Edge createEdge(String prefix, Long id, String name, String label) {
    var edge = new Edge(prefix + id.toString(), name);
    edge.setColor("black");
    edge.setArrows(new Arrows(new ArrowHead(1, Arrows.Type.arrow)));
    edge.setLength(300);
    return edge;
}

private Node createNode(String prefix, Long id, String name, String color) {
    var node = new Node(prefix + id, name);
    node.setShape(Shape.circle);
    node.setColor(color);
    node.setFont(Font.builder().withColor("white").build());
    node.setWidthConstraint(new WidthConstraint(100, 100));
    return node;
}

Finally, the graph is displayed in the application.

Conclusion

The application is still in an early stage. The graph will be extended and the diagram must be improved. Especially the behavior when dragging around the edges seems to be quite tricky and vis.js provides a lot of configuration.

As a Java developer creating UIs with Vaadin makes it very efficient. There are even 3rd party libraries that wrap components in a Java API. On the other side, I was impressed by how easy it is to start with Neo4j and to integrate it into a Spring Boot application.

Btw. If you want to learn more about Spring Boot check my video below.

Vaadin and jOOQ: Match Made in Heaven

Have you ever had to write an application that didn’t do much other than display and change data? And did you use a Single Page Application framework like Angular, a REST API, and Hibernate?

How about if that was a lot easier?

Introducing Vaadin and jOOQ

Vaadin exists already for 20 years but was completely overhauled three years ago. Vaadin Flow is a web framework that allows the development of a web application in Java. It’s based on web components and the state is synchronized between browser and server.

What makes Vaadin a perfect match for data-centric applications are the components. We will see how simple it is to display data in a grid.

The second framework we will use is jOOQ for database access. jOOQ consists of two parts: a code generator and a DSL.

With the code generator, you can generate Java code for all database objects including tables, views, procedures, user-defined types, and many more. The DSL allows you to write type-safe, compile-time-checked SQL in Java using the generated Java classes.

Displaying a List of Data

Data Providers

To fetch data from a backend Vaadin provides data providers. Data providers are used when a component displays a collection of data. For example, in Grids or ComboBoxes.

There are two types of data providers: InMemoryDataProvider and CallbackDataProvider. An InMemoryDataProvider is used if you only have a handful of records to display whereas the CallBackDataProvider provides lazy loading and paging.

Creating the Grid

We want to create a grid to display the revenue of customers for that we use a Java Record:

public record CustomerInfo(Integer id, String lastname, String firstname, double revenue) {
}

This Java Record can then be used to define the type of the grid and add the columns. The addColumn is an overloaded method and we use to pass a method reference that returns the value that must be displayed in the grid cell.

Grid<CustomerInfo> grid = new Grid<>();
grid.addColumn(CustomerInfo::id).setHeader("ID");
grid.addColumn(CustomerInfo::firstname).setHeader("First Name");
grid.addColumn(CustomerInfo::lastname).setHeader("Last Name");
grid.addColumn(CustomerInfo::revenue).setHeader("Revenue");

Fetching the Data

As we have many customers we want to use paging. The Grid has an overloaded method setItems where we only have to pass a callback that is used to fetch the data. Internally a CallbackDataProvider is created.

grid.setItems(query -> dsl
	.select(CUSTOMER.ID, CUSTOMER.LASTNAME, CUSTOMER.FIRSTNAME, DSL.sum(PRODUCT.PRICE))
	.from(CUSTOMER)
	.join(PURCHASE_ORDER).on(PURCHASE_ORDER.CUSTOMER_ID.eq(CUSTOMER.ID))
	.join(ORDER_ITEM).on((ORDER_ITEM.ORDER_ID.eq(PURCHASE_ORDER.ID)))
	.join(PRODUCT).on((PRODUCT.ID.eq(ORDER_ITEM.PRODUCT_ID)))
	.groupBy(CUSTOMER.ID)
	.orderBy(CUSTOMER.LASTNAME, CUSTOMER.FIRSTNAME)
	.offset(query.getOffset())
	.limit(query.getLimit())
	.fetchStreamInto(CustomerInfo.class)

To fetch the data we use the jOOQ DSL with the generated Java code. There are Java classes for the tables (CUSTOMER, PURCHASE_ORDER, ORDER_ITEM, and PRODUCT) that contain the columns (CUSTOMER.ID, CUSTOMER.LASTNAME, etc). Finally, we fetch the result into the CustomerInfo record.

We use offset and limit from the query object that is passed to the fetch callback method from Vaadin so the grid will have infinite scrolling and will call the callback method to fetch data when needed.

As you see in the example the code looks like SQL but due to the generated code, it is fully type-safe and is checked at compile time. If you have breaking database changes your code may no longer compile after regenerating the Java code.

And finally, this is what the grid looks like. There are many more features in grid-like filtering and sorting. You can find more information about these features and also about all other Vaadin components in the components directory.

Conclusion

This introduction covered only a small part of the feature set of both frameworks but I hope the example gives you an impression of how easy it is to create data-centric web applications with the combination of Vaadin and jOOQ.

If you’d like to know more check out the jOOQ documentation and tutorials and watch my Vaadin quick start tutorial to learn more about Vaadin:

Vaadin Tip: Lazy Loading and Item Identity

When using grids, trees, or any other of the multi-valued component with Vaadin you often want to display data from a database table, and typically you have more than a few rows in the database.
In this case loading, thousands or even millions of records don’t make sense and would be a huge performance problem. For this use case, Vaadin provides lazy loading using a CallbackDataProvider.

To create a CallBackDataProvider you must implement a CountCallback and a FetchCallback.
The CountCallback is used to provide the total number of records. And the FetchCallback is used for paging. Both methods receive an Query object that contains filter, sorting, offset and limit.

In this example, you can see how to use offset and limit.

DataProvider<Employee, Void> dataProvider = new CallbackDataProvider<>(
                query -> employeeRepository.findAll(query.getOffset(), query.getLimit()),
                query -> employeeRepository.count()
        );

Item Identity

In a Grid or the DataProvider there are methods that are using an item:

grid.select(employee);
dataProvider.refreshItem(employee);

Ever wondered how Vaadin is finding the right item in the underlying data structure? No surprise – it uses equals().
But what if you can’t control how equals() is implemented? For example, is the Class that you use in the Grid is generated directly from the database tables like jOOQ does?

No worries! Vaadin provides another constructor to create a CallbackDataProvivder

As a third parameter, you pass a ValueProvider that is responsible to return a unique identifier. In the example, this is the ID of the Employee.

DataProvider<Employee, Void> dataProvider = new CallbackDataProvider<>(
                query -> employeeRepository.findAll(query.getOffset(), query.getLimit()),
                query -> employeeRepository.count(),
                Employee::getId
        );

What’s next?

Ever heard of Vaadin? Stay tuned there will be a Vaadin introduction coming soon!

Type Safe SQL in Java

No matter if you are using frameworks like JPA, MyBatis or Spring Data JDBC you always end up declaring the SQL statements as a Java String.

Strings, String, Strings

No matter if you are using frameworks like JPAMyBatis or Spring Data JDBC you always end up declaring the SQL statements as a Java String.
The problem with this approach is that you have to write tests for every statement to make sure that it is even valid SQL. There is no compile time guarantee that the SQL statement will execute.

Get rid of the Strings!

Embedded SQL

I started professional software development in 1995 on IBM mainframe computers programming in COBOL. To access the database we used something called “Embedded SQL”:

EXEC SQL
SELECT lastname, firstname
INTO :lastname, :firstname
FROM employee
WHERE id = :id

The cool thing about Embedded SQL was that a pre-compiler was checking every SQL statement and only if it was valid the code compiled.
Bellow you can see the compile steps. (Source: http://www.redbooks.ibm.com/redbooks/pdfs/sg246435.pdf)

SQLJ

When I first met Java and JDBC in 2000 I was confused that nothing similar existed. I found out that there was an initiative called SQLJ started in 1997 but never took off. I don’t have an idea why, maybe because this was hard to integrate for IDE vendors and pre-compilers where not very common for Java. At least the compile steps are similar to Embedded SQL:

When comparing JDBC and SQLJ we can see that there is not much difference from the amount of code you have to write but everything after #sql is type safe because the pre-compiler checks the syntax where as with JDBC there is a String that could contain any error and the error will happen late in production.

And then I found jOOQ!

Ten years ago Lukas Eder release the first version of jOOQ. According to the website is jOOQ “The easiest way to write SQL in Java”

Let’s try to write the same query as above with jOOQ:

List<EmployeeDTO> records = create
         .select(EMPLOYEE.LASTNAME, EMPLOYEE.FIRSTNAME, EMPLOYEE.SALARY)
         .from(EMPLOYEE)
         .where(EMPLOYEE.SALARY.between(80000, 100000))
         .fetchInto(EmployeeDTO.class);

Pretty cool, isn’t it? Yes – but how does it work?

1. Code Generator

jOOQ uses a code generator to generate Java classes from database objects.

For example this is an extract of the class generated by jOOQ for the table EMPLOYEE:

public class Employee extends TableImpl<EmployeeRecord> {

    public static final Employee EMPLOYEE = new Employee();

    public final TableField<EmployeeRecord, Integer> ID = createField("ID", org.jooq.impl.SQLDataType.INTEGER.nullable(false).identity(true), this, "");
    public final TableField<EmployeeRecord, String> LASTNAME = createField("LASTNAME", org.jooq.impl.SQLDataType.VARCHAR(50).nullable(false), this, "");
    public final TableField<EmployeeRecord, String> FIRSTNAME = createField("FIRSTNAME", org.jooq.impl.SQLDataType.VARCHAR(50).nullable(false), this, "");
    public final TableField<EmployeeRecord, Integer> SALARY = createField("SALARY", org.jooq.impl.SQLDataType.INTEGER, this, "");
    public final TableField<EmployeeRecord, Integer> DEPARTMENT_ID = createField("DEPARTMENT_ID", org.jooq.impl.SQLDataType.INTEGER.nullable(false), this, "");
    public final TableField<EmployeeRecord, Integer> MANAGER_ID = createField("MANAGER_ID", org.jooq.impl.SQLDataType.INTEGER, this, "");
}

There are constants for the table and all the columns. Thanks to these meta data classes it’s not possible to use a type in a SQL statement that does not exists in the database. And because you can generate the meta data every time, the database model changes your code will not compile if there are breaking changes.

How to configure the generator and what input formats for the generator are possible will be described in a future post. (Stay tuned)

2. Domain Specific Language

The second part of jOOQ is the DSL (Domain Specific Language) that allows to write SQL code in Java.
And in contrast to SQL in Strings the DSL forces me to write valid SQL!

Examples

So let’s see some more examples. The examples are based on this data model:

Insert
dsl.insertInto(DEPARTMENT)
   .columns(DEPARTMENT.NAME)
   .values("HR")
   .execute();
Select
dsl.select(DEPARTMENT.NAME)
    .from(DEPARTMENT)
    .where(DEPARTMENT.NAME.eq("IT"))
    .fetchOne();
Update
dsl.update(DEPARTMENT)
   .set(DEPARTMENT.NAME, "IT2")
   .where(DEPARTMENT.ID.eq(departmentId))
   .execute();
Delete
dsl.deleteFrom(EMPLOYEE)
   .where(EMPLOYEE.ID.eq(employeeId))
   .execute();

What’s next?

That was just a short introduction. In the next blog post we will have a deeper look at all the features jOOQ provides.

In the meanwhile you can checkout the code here: https://github.com/simasch/jooq-hr