GraphLake Documentation

Introduction

GraphLake is a graph database that supports a unified OneGraph model, seamlessly combining property graphs and RDF. This means you do not need to choose between the two graph types; it supports both simultaneously, offering flexibility and versatility for a wide range of use cases.

Designed for scalability and performance, GraphLake supports both large analytical workloads and Online Transaction Processing (OLTP) scenarios. Its architecture draws inspiration from the MPP style of systems like Apache Iceberg and DeltaLake. However, GraphLake introduces automatic data partitioning, with representation and filtering mechanisms specifically optimized for graph data structures.

GraphLake is engineered to work efficiently with files stored locally on high-performance NVMe or SSD drives or in cloud storage solutions such as Amazon S3 (and S3-compatible options like MinIO) or Azure Blob Storage. The system allows independent scaling of compute and storage resources to match workload requirements. For instance, you can store large volumes of data cost-effectively while using a small compute instance, or opt for a configuration with smaller datasets and larger compute resources to support high-performance querying.

Leveraging parallel processing and memory, GraphLake delivers impressive performance, with the ability to scale hardware resources for even greater efficiency.

GraphLake is not offered as a managed service. Instead, it is designed to be deployed within your cloud environment or run on your hardware, giving you complete control over your setup and infrastructure. We provide a developer edition that is free to use indefinitely, making it easy to get started with GraphLake and explore its capabilities.

Versions

There are 4 versions of GraphLake

• Developer Edition - free for devs forever
• Business - license required
• Business Pro - license required
• Enterprise - license required

Installation

GraphLake developer edition is available now, commercial releases will be coming soon. If you would like to know more or start using GraphLake commercially, please email "contact @ dataplatformsolutions.com"

GraphLake is available as a docker image. The docker image is available here on docker hub.

Downloads

• Download the latest docker image

Either download and unpack the binary from the .zip package and place it on your path, or pull the docker image:

docker pull dataplatformsolutions/graphlake:latest

Cloud Deployments

To support deploying GraphLake into your cloud we have provided example terraform scripts that can be used and adapted to a specific setup. Deployment of GraphLake through the AWS and Azure marketplace is coming soon.

AWS

AWS terraform on https://github.com/dataplatformsolutions/graphlake-deploy

Available in the AWS marketplace - coming soon!

Azure

Azure terraform on https://github.com/dataplatformsolutions/graphlake-deploy

Available in the Azure marketplace - coming soon!

Digital Ocean

Digital Ocean terraform on https://github.com/dataplatformsolutions/graphlake-deploy

Config and Startup

GraphLake supports the following startup config environment variables:

To run GraphLake in a Docker container as a detached process, passing the environment variables and mapping a local folder to the configured data path, you would run:

docker run -d \
      -e GRAPHLAKE_BACKEND_TYPE=local \
      -e GRAPHLAKE_STORE_PATH=/data \
      -e GRAPHLAKE_IMPORT_LOCATION=import \
      -v /path/to/local/data:/data \
      dataplatformsolutions/graphlake:latest

API

Query

POST /stores/:store/query

Execute a query in the specified :store. GraphLake supports three different query langauges: SPARQL (core subset), SemanticCypher, and GraphLake Javascript. See the sections in the query section for details on each langauge. Use the following content types to specify which query you are using: application/sparql, application/x-graphlake-query-semanticcypher, application/x-graphlake-query-javascript.

POST /stores/myStore/query
Content-Type: text/plain

SELECT ?s ?p ?o WHERE { ?s ?p ?o }
            

HTTP/1.1 200 OK
Content-Type: application/json

{
  "results": [
    {
      "s": "http://example.org#subject1",
      "p": "http://example.org#predicate1",
      "o": "http://example.org#object1"
    },
    ...
  ]
}
            

Import Data into a Graph

POST /stores/:store/graphs/:graph/import

Import data (in N-Triples format, or CSV) into a specified :graph. To import csv as nodes use the content type

application/x-graphlake-node+csv

application/x-graphlake-node+csv

POST /stores/myStore/graphs/myGraph/import
Content-Type: application/n-triples

<http://example.org#subject> <http://example.org#predicate> <http://example.org#object> .
            

Create Graph

POST /stores/:store/graphs/:graph

Create an empty graph in the specified :store.

HTTP/1.1 200 OK
{
  "message": "Graph created"
}
            

List Graphs

GET /stores/:store/graphs

Retrieve a list of available graphs in the specified :store.

HTTP/1.1 200 OK
Content-Type: application/json

[
  "graph1",
  "graph2",
  ...
]
            

Delete Graph

DELETE /stores/:store/graphs/:graph

Delete a specific graph within a store.

HTTP/1.1 200 OK
{
  "message": "Graph deleted"
}
            

Create Store

POST /stores/:store

Create a new store.

HTTP/1.1 200 OK
{
  "message": "Store created"
}
            

List Stores

GET /stores

Retrieve a list of all stores. If security is enabled, only stores accessible to the current user are returned.

HTTP/1.1 200 OK
[
  "store1",
  "store2",
  ...
]
            

Delete Store

DELETE /stores/:store

Delete a specified store. This operation is irreversible.

HTTP/1.1 200 OK
{
  "message": "Store deleted"
}
            

Add User

POST /admin/users

Create a new user in the system.

{
  "username": "testuser",
  "password": "secretpassword",
  "public_key": "-----BEGIN PUBLIC KEY-----..."
}
            

HTTP/1.1 200 OK
{
  "message": "User added"
}
            

Delete User

DELETE /admin/users/:username

Delete an existing user by username.

HTTP/1.1 200 OK
{
  "message": "User deleted"
}
            

Generate Key Pair for a User

POST /admin/users/:username/keypair

Generate a new private/public key pair for the specified user.

HTTP/1.1 200 OK
{
  "private_key": "-----BEGIN PRIVATE KEY-----...",
  "public_key": "-----BEGIN PUBLIC KEY-----..."
}
            

Authenticate with Password

POST /admin/authenticate/password

Obtain a JWT token by providing username and password.

{
  "username": "testuser",
  "password": "secretpassword"
}
            

HTTP/1.1 200 OK
{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
            

Authenticate with JWT

POST /admin/authenticate/jwt

Verify an existing JWT and obtain a renewed token.

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}
            

HTTP/1.1 200 OK
{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9... (new token)"
}
            

Set User Rules

POST /admin/users/:username/rules

Define or update the security rules for a specified user.

[
  {
    "resource": "/stores/myStore/graphs/graph1",
    "permission": "read"
  },
  {
    "resource": "/stores/myStore",
    "permission": "write"
  }
]
            

HTTP/1.1 200 OK
{
  "message": "Rule added"
}
            

CURL Examples

    # Example: Create a store and a graph, then import data.

    # 1. Create a new store
    curl -X POST http://localhost:8080/stores/myNewStore
    
    # 2. Create a new graph in that store
    curl -X POST http://localhost:8080/stores/myNewStore/graphs/myGraph
    
    # 3. Import N-Triples data from a file
    curl -X POST http://localhost:8080/stores/myNewStore/graphs/myGraph/import \
         -H "Content-Type: text/plain" \
         --data-binary @path/to/your/file.nt
    
    # 4. Query the data
    curl -X POST http://localhost:8080/stores/myNewStore/query \
         -H "Content-Type: text/plain" \
         --data "SELECT ?s ?p ?o WHERE { ?s ?p ?o }"

Data Import

Query

SPARQL

GraphLake supports a subset of SPARQL features for querying RDF data. This document outlines the supported features, including query forms, patterns, and functions.

Query Forms

1. SELECT
   • Description: Retrieves variables bound in a query pattern match.
   • Example:

   
   SELECT ?subject ?predicate ?object
   WHERE {
     ?subject ?predicate ?object.
   }
                   

2. ASK
   • Description: Returns a boolean indicating whether a query pattern matches.
   • Example:

   
   ASK WHERE {
     ?subject ?predicate ?object.
   }
                   

Patterns

1. Triple Patterns
   • Description: Basic graph pattern matching triples.
   • Example:

   
   SELECT ?subject ?object
   WHERE {
     ?subject  ?object.
   }
                   

2. Group Graph Patterns
   • Description: A set of triple patterns enclosed in braces.
   • Example:

   
   SELECT ?subject ?object
   WHERE {
     {
       ?subject  ?object.
     }
     UNION
     {
       ?subject  ?object.
     }
   }
                   

3. Optional Patterns
   • Description: Patterns that may or may not match.
   • Example:

   
   SELECT ?subject ?object ?optionalObject
   WHERE {
     ?subject  ?object.
     OPTIONAL { ?subject  ?optionalObject. }
   }
                   

4. Filter Patterns
   • Description: Restricts solutions based on a boolean expression.
   • Example:

   
   SELECT ?subject ?object
   WHERE {
     ?subject  ?object.
     FILTER (?object > 10)
   }
                   

Functions

1. Bound
   • Description: Checks if a variable is bound.
   • Example:

   
   SELECT ?subject
   WHERE {
     ?subject  ?object.
     FILTER (BOUND(?object))
   }
                   

2. isIRI
   • Description: Checks if a term is an IRI.
   • Example:

   
   SELECT ?subject
   WHERE {
     ?subject ?predicate ?object.
     FILTER (isIRI(?subject))
   }
                   

3. isLiteral
   • Description: Checks if a term is a literal.
   • Example:

   
   SELECT ?subject
   WHERE {
     ?subject ?predicate ?object.
     FILTER (isLiteral(?object))
   }
                   

4. isBlank
   • Description: Checks if a term is a blank node.
   • Example:

   
   SELECT ?subject
   WHERE {
     ?subject ?predicate ?object.
     FILTER (isBlank(?subject))
   }
                   

5. STR
   • Description: Returns the string representation of a term.
   • Example:

   
   SELECT ?subject (STR(?object) AS ?objectStr)
   WHERE {
     ?subject ?predicate ?object.
   }
                   

6. LANG
   • Description: Returns the language tag of a literal.
   • Example:

   
   SELECT ?subject (LANG(?object) AS ?lang)
   WHERE {
     ?subject ?predicate ?object.
   }
                   

7. DATATYPE
   • Description: Returns the datatype IRI of a literal.
   • Example:

   
   SELECT ?subject (DATATYPE(?object) AS ?datatype)
   WHERE {
     ?subject ?predicate ?object.
   }
                   

8. CONTAINS
   • Description: Checks if a string contains another string.
   • Example:

   
   SELECT ?subject
   WHERE {
     ?subject ?predicate ?object.
     FILTER (CONTAINS(STR(?object), "example"))
   }
                   

9. STRSTARTS
   • Description: Checks if a string starts with another string.
   • Example:

   
   SELECT ?subject
   WHERE {
     ?subject ?predicate ?object.
     FILTER (STRSTARTS(STR(?object), "http://"))
   }
                   

10. STRENDS
    • Description: Checks if a string ends with another string.
    • Example:

    
    SELECT ?subject
    WHERE {
      ?subject ?predicate ?object.
      FILTER (STRENDS(STR(?object), ".org"))
    }
                    

GraphLake's SPARQL support includes essential query forms, patterns, and functions to perform effective graph data queries. Use the provided examples to construct and execute your SPARQL queries.

SemanticCypher

This is still being developed but in essence it is OpenCypher but we introduce a namespace directive and then change the form of variables and identifiers, to be gra?things:Person rather than cypher with gra:Person. There are also some built in properties like __id, which is the subject URI of a node.

GraphLake JS

GraphLake provides a JavaScript-based query language to interact with the graph data. This language allows you to perform various operations such as matching triples, committing transactions, and writing query results.

GraphLake executes JS and inserts into the runtime two objects,

_context

_writer

Match Triples

_context.matchTriples(subject, predicate, object, datatype, isLiteral, graphs)

Description: Matches triples in the specified graphs.

Parameters:

• subject (string): The subject of the triple.
• predicate (string): The predicate of the triple.
• object (string): The object of the triple.
• datatype (string): The datatype of the object.
• isLiteral (boolean): Whether the object is a literal.
• graphs (array of strings): The graphs to search in.

Returns: An iterator for the matched triples.

Assert Triple

_context.assertTriple(subject, predicate, object, datatype, isLiteral, graph)

Description: Asserts a new triple in the specified graph.

Parameters:

• subject (string): The subject of the triple.
• predicate (string): The predicate of the triple.
• object (string): The object of the triple.
• datatype (string): The datatype of the object.
• isLiteral (boolean): Whether the object is a literal.
• graph (string): The graph to insert into.

Commit Transaction

_context.commit()

Description: Commits the current transaction.

Returns: true or false if the transaction was successful.

Delete Triple

_context.deleteTriple(subject, predicate, object, datatype, isLiteral, graph)

Description: Deletes a triple from the specified graph.

Parameters:

• subject (string): The subject of the triple.
• predicate (string): The predicate of the triple.
• object (string): The object of the triple.
• datatype (string): The datatype of the object.
• isLiteral (boolean): Whether the object is a literal.
• graph (string): The graph to delete from.

Write Header

_writer.writeHeader(headers)

Description: Writes the header for the query result.

Parameters:

• headers (array of strings): The header columns.

Returns: Undefined.

Write Row

_writer.writeRow(row)

Description: Writes a row to the query result.

Parameters:

• row (array of any): The row data.

Returns: Undefined.

Examples

  _writer.writeHeader(["Subject", "Predicate", "Object"]);
  
  let triplesIter = _context.matchTriples("http://example.org/subject", "", "", "", true, ["graph1"]);
  while (true) {
      let triple = triplesIter.next();
      if (triple == null) {
          break;
      }
      _writer.writeRow([triple.subject, triple.predicate, triple.object]);
  }
      

Explanation: This script matches all triples with the subject http://example.org/subject in graph1 and writes the results to the output.

  // Delete an existing triple
  _context.deleteTriple("http://example.org/subject", "http://example.org/predicate", "http://example.org/object", "", false, "graph1");

  // Add two new triples
  _context.assertTriple("http://example.org/subject1", "http://example.org/predicate1", "http://example.org/object1", "", false, "graph1");
  _context.assertTriple("http://example.org/subject2", "http://example.org/predicate2", "http://example.org/object2", "", false, "graph1");

  // Commit the transaction
  _context.commit();
      

Explanation: This script commits the current transaction.

  _writer.writeHeader(["Id", "Name"]);
  
  let triplesIter = _context.matchTriples("http://example.org/person", "http://example.org/name", "", "", true, ["graph1"]);
  while (true) {
      let triple = triplesIter.next();
      if (triple == null) {
          break;
      }
      _writer.writeRow([triple.subject, triple.object]);
  }
      

Explanation: This script writes a header with columns "Id" and "Name", matches triples with the subject http://example.org/person and predicate http://example.org/name in graph1, and writes the results to the output.

  _writer.writeHeader(["Subject", "Predicate", "Object"]);
  
  let graphs = ["graph1", "graph2"];
  let triplesIter = _context.matchTriples("http://example.org/subject", "", "", "", true, graphs);
  while (true) {
      let triple = triplesIter.next();
      if (triple == null) {
          break;
      }
      _writer.writeRow([triple.subject, triple.predicate, triple.object]);
  }
      

Explanation: This script matches all triples with the subject http://example.org/subject in both graph1 and graph2 and writes the results to the output.

Security

GraphLake uses signed JWT to allow access to the API. Tokens are fetched from the API by authenticating with either a signed JWT token using a local private key or user name and password. Users are managed on the server to give read, write and owner access to stores and graphs.

When a security manager is enabled, certain endpoints require a valid JWT token for authorization. The server applies ACL checks to ensure that a user can only access the stores and graphs permitted by their assigned rules. If you receive a 401 (Unauthorized) or 403 (Forbidden) response, check that:

• Your JWT is valid and has not expired.
• You have the necessary permissions (read/write) for the requested resource.

The developer edition runs unsecured.

Monitoring

GraphLake writes structured logs to StdErr which when run in docker goes into the docker logs. We recommend shipping these logs to your favourite log management system and configuring any notifications you require.

VSCode client

To enable a great developer experience we provide a VSCode extension for managing stores and graphs, running queries, and user managemment. All operations can also be done from CURL or programtically over http in any language.

Installation

Support

If you have issues with GraphLake then please reach out to us on discord, file an issue on public github repository, or send an email to "contact @ dataplatformsolutions.com".