Skip to main content

· 7 min read

What's New

Version 2.2 includes following new features and changes.

Graph Security Layer

New features to enable additional filtering and automatic Access Rule generation were added to Graph Security Layer.

Custom ExternalGraphSecurity workers allow to additionally filter graph related results after Access Rules are applied. Once registered in GBS, each graph related call to EBS will also call ExternalGraphSecurity workers which may then filter results based on customer specific systems or rules. For more information regarding the implementation of such a worker, refer to the developer documentation.

Since manually creating and maintaining Access Rules can be cumbersome, version 2.2 adds the possibility to automatically generate Access Rules based on external systems. A new Rule Transformation step can be added to Load Plans. This step allows to define a registered RuleTransformation worker that is called when the step is executed. Such a worker can be used to automatically create Access Rules based on customer specific systems. The data a worker needs to create rules (e.g., users, entities, properties, etc.) can be defined in the Rule Transformation step. For more information regarding the implementation of such a worker, refer to the developer documentation.

caution

In order for rule transformation steps to work in Airflow, the service account of explore-airflow client needs additional roles assigned in Keycloak:

  • xpl-gbs-target-entities-execute (realm role)
  • xpl-gbs-worker-framework-execute (realm role)
  • view-users (role of realm-management client)

An active flag was added to workers in GBS which allows to pause the usage of a worker in the system. This comes in handy if for example an ExternalGraphSecurity worker is malfunctioning or not available. By pausing this specific worker the system will still work instead of returning errors because of a faulty worker.

Until now it was assumed that workers only provide GET / and POST / endpoints. Because of this, GBS took the provided worker URL as is to get basic information when adding a new worker to the system. With the new version custom endpoints are supported and GBS ensures to call GET / when registering a new worker no matter which worker URL is provided.

The way property value conditions are displayed in an Access Rule has been improved. Instead of displaying each individual condition in a row, they are now grouped by entity and property.

With version 2.2 EBS' /api/graph/query/cypher endpoint is deprecated. This endpoint allowed to execute any cypher query and directly returned the result from Neo4j which made it impossible to apply Graph Security Layer to any results. The endpoint is superseded by /api/graph/cypher/count and /api/graph/cypher/nodes. These endpoints also allow to execute custom cypher queries but only return a count result respectively nodes contained in a result. Additionally, the latter also applies Graph Security Layer.

Data Pump Overrides

The way how Data Pump overrides work was changed to be more resilient and performant. Until now all overrides were executed at once. If three overrides were defined, GBS collected data of all overrides before processing them. This could lead to failures while importing because of duplicated entries and impact performance of the system. With version 2.2, overrides are handled individually. Each override will be it's own task in Airflow and therefore processed independent of each other.

Additionally, support of placeholders was added to overrides which allows to define dates passed to overrides (see Load Plan documentation for more information). This can, for instance, be used to pass a date to an endpoint in order to only retrieve data of a certain point in time.

Frontend Graph Layout and Performance Improvements

Besides overall graph performance improvements, following changes have been made to improve the look and feel when working with the Frontend:

  • Replace hierarchical layout algorithm by a more performant one
  • Add top down order to by-level layout
  • Decrease distance between nodes for by-level layout
  • Add lasso selection that allows to freely select multiple nodes
  • Add performance indicator for layouts
  • Add warning indicator when too much nodes are displayed and display a hint to change the layout
  • Add suggestion to change layout in "too many nodes" warning
  • Focus and select newly added nodes in sandbox mode
  • Add option to toggle automatic centering/fitting of nodes after running layout
  • Add min and max zoom limit

Removal of [Empty] Expression

Data Importer allowed to map an [Empty] expression to any property. This predefined expression was removed since it is also possible to define an empty string with through "Set expression".

Improvements

  • [Frontend] Menu for saving sessions was simplified
  • [Frontend] A "quick save" button was added to top menu
  • [Frontend] A "delete session" option was added to the top menu
  • [Frontend] "Remove selected" was added to the default options of the node menu
  • [Frontend] Added error screens for missing nodes in sessions and viewpoints
  • [Frontend] Added detailed tooltips for graph controls
  • [Frontend] Added "Provide" menu to graph overview
  • [Frontend] Added cancel pending request dialog when loading takes too long
  • [Frontend] Improved interaction between label and property filters
  • [Frontend] Import sandbox functionality was moved from export to top menu
  • [Frontend] Webvis library was updated to work with 3DHub 3.7
  • [Frontend] Overall performance improvements
  • [GBS] Loading data of dashboard tabs is done lazily when clicking on tabs instead of greedily when opening GBS UI
  • [GBS] Use same date formats throughout the dashboard
  • [GBS] Removed superfluous "Smoke test" button in Data Pump containers
  • [GBS] Updated "Clear relationship" icon to make clear that it triggers a destructive behavior
  • [GBS] When clicking on "Add related" in graph processing, all related entities are added instead of only related entities in selected workspace

Bug Fixes

  • [Frontend] Node menu is not immediately closed when clicking "expand"
  • [Frontend] "Expand" menu might get stuck when opening node menu
  • [Frontend] Edges won't reset color after selection in graph overview
  • [Frontend] Cytoscape occasionally freezes in Firefox when zooming
  • [Frontend] Authentication failed view cannot be rendered because theme is not present
  • [Frontend] Filter by label is not working as expected
  • [Frontend] Dashboard is crashing when roles are missing
  • [Frontend] Property filter is not sorted correctly when toggling labels
  • [Frontend] When using a property filter on a deselected entity, other entities cannot be selected
  • [Frontend] When saving a renamed session, the old name is displayed shortly before menu is closed
  • [GBS] When relationship diagram is moved to the bottom of the page it is rendered on top of the session logs
  • [GBS] Changing a property aggregation while entity filter is active breaks filtering
  • [GBS] Fixed a few bugs that could appear when creating multiple entities from a container
  • [GBS] Dashboard does not consider selected workspaces when opened for the first time
  • [GBS] First startup of a new instances fails because not correctly updating internal state
  • [GBS] Queue jobs table in dashboard overlaps session logs
  • [GBS] In graph processing it is not possible to select a load type if build type was selected before an entity
  • [GBS] Entities belonging to a selected workspace are not displayed in property aggregation drop-down
  • [GBS] Search index creation fails when empty entity is processed
  • [GBS] It is possible to delete files from Data Pump image and save it without any files

Deprecations

  • [EBS] /api/graph/query/cypher has been deprecated and replaced by /api/graph/cypher/count and /api/graph/cypher/nodes endpoints. The deprecated endpoint will be removed in the next release.

· 5 min read

What's New

Version 2.1 includes following new features and changes.

The index that was used for full-text search in older versions was prone to break on very big datasets. In Version 2.1 there is a new way how full-text search indexes are created and queried. Future releases will include further performance improvements of the search.

caution

Existing data has to be reprocessed to make them available in the search.

Performance Improvement of "Merge" Data Import

The performance of data imports using "Merge" load type was improved drastically. While importing data for an entity with 300.000 nodes and 15 million relationships took 2 hours and 38 minutes with the old logic, the same entity now only takes 6 minutes to import.

GBS Dashboard

Version 2.1 introduces a new dashboard in the Graph Builder Services. The data previously available on the dashboard was split into different views. Additionally, an overview of the whole system was added to the dashboard.

Telemetry

OpenTelemetry, Jaeger and Prometheus are added to the stack and integrated in services to trace requests across the stack. This helps to track usage of components and identify bottlenecks in communication between services.

New Way of Storing Relationships

When importing data, all corresponding relationships are precalculated and stored in the system. The way how these relationships are stored has been changed which results in a performance improvement. In the past it could take a long time to rename a property that is part of a relationship (~4 minutes for 15 million relationships). With the new approach the same rename only takes ~2 minutes and 30 seconds. In the course of these changes obsolete artifact types " PartID" and "RelatedNodePath" were removed.

Improvements

  • [Frontend] Consolidated filtering and removing nodes in sandbox
  • [Frontend] Performance improvement when retrieving nodes in a Data Lab
  • [Frontend] Improved error messages
  • [Frontend] Filter button is permanently visible in top navigation bar
  • [Frontend] Cleaned up of left navigation bar
  • [Frontend] Added new options for context menu
  • [Frontend] Allow easy reset to defaults of different personalization settings
  • [Frontend] Result table in add related and search menu allows to select different properties to inspect data
  • [Frontend] It is possible to personalize icons for labels
  • [Frontend] Consolidated left menu in sandbox mode with menu in exploration mode
  • [Frontend] Settings button was added to node context menu to easily access context menu settings
  • [Frontend] "Property selection" window was moved from the right menu bar to a dialog opened through setting button in properties panel
  • [Frontend] In "Filter by property" the global "Keyword" search was replaced by "Search all properties"
  • [Frontend] Filter history now includes what was filtered for
  • [Frontend] Active filters are displayed as tags to the filter menu
  • [Frontend] Webvis library was updated to work with 3DHub 3.6
  • [Frontend] Performance improvements by replacing calls to the now deprecated /api/graph/query/nodes endpoint
  • [GBS] List of artifact definitions in target entities now also displays the artifact type [GBS] A warning will be displayed in data importer when selecting a target entity whose connected source entity is missing
  • [GBS] Added explanation about order for discrete values
  • [GBS] When a property, that is part of a relationship, is deleted, the corresponding relationship is also deleted
  • [GBS] When creating an entity from a container, excluded properties are not shown in builders anymore
  • [GBS] Separators are added to large numbers
  • [GBS] System configuration now requires admin role
  • [GBS] In Access Rules, when selecting an already added entity, already added properties will automatically be selected
  • [GBS] "DB Query" and "Graph Query" were removed for security reasons
  • [GBS] Future dates cannot be selected in Queue Jobs or Logs anymore

Bug Fixes

  • [Frontend] Open new session button in the Data Lab does not open a new tab
  • [Frontend] When switching tabs, correlations in Data Lab are scaled down
  • [Frontend] After applying a scope, the toggle is malfunctioning
  • [Frontend] Discrete values are not correctly handled in Data Lab charts
  • [Frontend] Date and range filter are malfunctioning
  • [Frontend] Adding a rule group to a scope breaks the layout
  • [Frontend] Filter and select is not working for data with whitespaces or hyphens
  • [GBS] Opening the data table of an entity breaks layouts in other GBS views
  • [GBS] Properties used in Access Rules cannot be deleted
  • [GBS] Masked values in System Configuration cannot be edited
  • [GBS] Clicking "Create new entities" in Data Pump container twice, shows empty list
  • [GBS] Renaming an entity with properties ending on _ID, created superfluous properties
  • [GBS] Fixed multiple problems when creating entities from container
  • [GBS] Renaming a property that is used in multiple relationships causes an error
  • [GBS] Creating a relationship without selecting a property but typing its name leads to broken relationship

Deprecations

  • [EBS] /api/graph/query/nodes endpoint will be replaced by specific endpoints in future versions and should not be used anymore. For retrieving artifacts /api/artifacts/per-nodeid endpoint was added. Related nodes can be retrieved via already existing endpoints.

Deployment Changes

  • OpenTelemetry, Jaeger and Prometheus containers were added to enable tracing and metrics of communication between services in the stack.
  • All containers in the stack now run in non-root mode

Software Updates

3rd party software was updated to the following versions:

NameVersion
Neo4j5.5
Apache Airflow2.6
Keycloak21
OpenTelemetry0.76
Jaeger1.42
Prometheus2.42

· 4 min read

What's New

Version 2.0 includes following new features and changes.

Features

  • Worker Framework: The worker framework is the basis for using workers in the EDCH stack. Workers are isolated applications that are loosely integrated with GBS.
  • Expressions: Expressions allow the dynamic definition of property values. The required Expression Worker utilizes Python based Pandas to evaluate the expressions. Other use cases are the business key and artifacts builders which allow to define keys and artifacts based on expressions.
  • Graph Security Layer: The Graph Security Layer allows defining fine-grained rules for data visibility when retrieved via EBS endpoints. The rules are defined in GBS UI and are loaded by EBS. When EBS receives a graph related request it loads the corresponding rules and delegates filtering to a Graph Security specific worker.
  • GBS UI: GBS UI got some small updates to be more consistent throughout all pages for example by always having button descriptions and icons. Information texts have been moved to dedicated icons to save space for displaying data.
  • Import/Export Entities (Beta): A possibility to import and export target entities was added. It is possible to select which entities and dependent configurations should be exported. Exported entities can then be imported in any EDCH instance.
  • Performance & Stability: A lot of changes have been made to increase performance and stability of the whole stack.

Frontend

  • Redesign: The most prominent change in the Frontend is the complete UI redesign to have a modern look & feel. It features a new left menu structure and interaction as well as improved menus for searching, adding and filtering nodes. The graph interface was improved and offers additional layout options. The new dashboard allows to easily jump into different modes and open data you recently worked on.
  • Label View Layout: Performance improvements and automatic detection of sensitivity for each label view. The latter automatically sets the sensitivity to the nearest where related nodes can be found.
  • Personalisation: More customization settings for graph layout were added. It is also possible to set personalized colors for labels.
  • Data Lab: Data Lab was redesigned and a new charts library was integrated to further improve UI and interaction possibilities. It is possible to have multiple properties within one correlation and multiple property correlations next to each other. Important correlations can also be saved to the pinboard for later use or comparison.
  • Performance & Stability: Overall performance and stability has been improved.

Breaking Changes

Version 2.0 includes following breaking changes.

General

  • Naming conventions of Keycloak roles have changed. Therefore the license has to be re-uploaded once version 2.0 is deployed. By doing so, Keycloak roles will be recreated with correct naming.

GBS

  • Required roles on endpoints have been reworked and changed. Refer to GBS API's Swagger documentation for more information.

EBS

  • Membership related endpoints, which allowed to get and verify access tokens, where removed for security reasons. Applications requiring an access token should get one directly via Keycloak.
  • Support for parameters was removed from /api/graph/query/cypher endpoint for security reasons. Since it is not possible to apply the Graph Security Layer to this endpoint it is now additionally protected by xpl-admin-execute role.
  • Required roles on endpoints have been reworked and changed. Refer to EBS API's Swagger documentation for more information.

Deployment Changes

Starting with 2.0 it is highly recommended to deploy EDCH on a Kubernetes environment. This is mandatory for automatic scaling of workers. The stack still works with a docker compose deployment but performance will suffer.

Software Updates

3rd party software was updated to the following versions. A mandatory update indicates that EDCH is not compatible with an older version anymore.

NameVersionMandatory
PostgreSQL15.x
Neo4j5.x
RabbitMQ3.11.x
Apache Airflow2.5.x✔️
Keycloak20.x✔️