0

I generate graph data using Spark's GraphX library and I need an efficient way to store graph data. I have access to Apache Cassandra and ideally I want to store the graph data in there but the issue is I don't know how to efficiently store the graph structure in Cassandra. I have looked at Titan's backend storage in Cassandra but I could not find a detailed explanation of how the data (edges and properties) is formatted and stored. I am giving Titan as an example, any existing alternative or methodology is welcome. The persistence layer doesn't even have to be Cassandra necessarily. I mostly need an efficient way to store the graph that also enables me to do filtering on the backend. I.e. if I store on HDFS, I have to load the whole DB content on Spark to analyze it and perform transformations, and for big data that is unfeasible so I need a kind of storage that enables me to filter server-side what I need before loading it onto my memory.

Edit: My requirements is that this is a distributed graph database that can handle Big Data, is scalable and has to have a Spark connector of sorts (or a way to connect with Spark functionalities).

  • What about a Graph Database? – joanolo Jul 26 '17 at 20:05
  • You are right, I have looked into many of them but so far only Titan would have been suitable (if it hadn't been discontinued) and Janusgraph is still a long way to being a viable alternative. So I am looking either for an alternative to Titan or a methodology for storing graph data in Cassandra efficiently. – Mnemosyne Jul 26 '17 at 21:00
  • You can connect Neo4J to Spark... neo4j.com/blog/neo4j-3-0-apache-spark-connector – joanolo Jul 26 '17 at 21:18
1

TitanDB stores graphs in adjacency list format which means that a graph is stored as a collection of vertices with their adjacency list. The adjacency list of a vertex contains all of the vertex’s incident edges (and properties).

They used VertexID is the partition key, PropertyKeyID or EdgeID as clustering key and property value or edge properties as normal column.

Titan Data Layout

Their Cassandra Table schema :

CREATE TABLE edgestore (
    key blob,
    column1 blob,
    value blob,
    PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE AND CLUSTERING ORDER BY (column1 ASC);

Where

key     => VertexID
column1 => PropertyKeyID or DIR+OtherID+EdgeID
value   => PropertyValue or Pair of EdgePropertyKeyID and EdgePropertyValue
  • So in column1 they store the ID of the edges and then in value, they store properties of these edges? Am I understanding this correctly? – Mnemosyne Jul 31 '17 at 13:16
  • Yes. If the column1 is property key id then the value will be the property value of that key for the vertex and if it is Edge ID then the value will be properties of the edge – Ashraful Islam Jul 31 '17 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.