KVS: Abstract Erlang Database

KVS is an Erlang abstraction over various native Erlang key-value databases, like Mnesia. Its meta-schema includes only concept of iterators (persisted linked lists) that are locked or guarded by containers (list head pointers). All write operations to the list are serialized using a single Erlang process to provide sequential consistency. The application which starts Erlang processes per container called feeds.

The best use-case for KVS and key-value storages is to store operational data. This data should be later fed to SQL data warehouses for analysis. Operational data stores should be scalable, secure, fault-tolerant and available. That is why we store work-in-progress data in key-value storages.

KVS also supports queries that require secondary indexes, which are not supported by all backends. Currently KVS includes following storage backends: Mnesia, Riak and KAI.

Polymorphic Records

Any data in KVS is represented by regular Erlang records. The first element of the tuple as usual indicates the name of bucket. The second element usually corresponds to the index key field.

Rec = {user,"maxim@synrc.com",[]}. RecordName = element(1, Rec). Id = element(2, Rec).

Iterators

Iterator is a sequence of fields used as interface for all tables represented as doubly-linked lists. It defines id, next, prev, feed_id fields. This fields should be at the beginning of user’s record, because KVS core is accessing relative position of the field (like #iterator.next) with setelement/element BIF, e.g.

setelement(#iterator.next, Record, NewValue).

Iterator is a sequence of fields used as interface for all tables represented as doubly-linked lists. It defines id, next, prev, feed_id fields and should be in the begin of the record’s memory footprint because KVS core is accessing relative position of the field (like #iterator.next) with setelement/element BIF.

All records could be chained into the double-linked lists in the database. So you can inherit from the ITERATOR record just like that:

-record(access, {?ITERATOR(acl), entry_id, acl_id, accessor, action}).
#iterator { record_name, id, version, container, feed_id, prev, next, feeds, guard }

This means your table will support add/remove linked list operations to lists.

1> kvs:add(#user{id="mes@ua.fm"}). 2> kvs:add(#user{id="dox@ua.fm"}).

Read the chain (undefined means all)

3> kvs:entries(kvs:get(feed, user), user, undefined). [#user{id="mes@ua.fm"},#user{id="dox@ua.fm"}]

or just

4> kvs:entries(user). [#user{id="mes@ua.fm"},#user{id="dox@ua.fm"}]

Read flat values by all keys from table:

4> kvs:all(user). [#user{id="mes@ua.fm"},#user{id="dox@ua.fm"}]

Containers

If you are using iterators records this automatically means you are using containers. Containers are just boxes for storing top/heads of the linked lists. Here is layout of containers:

#container { record_name, id, top, entries_count }

Extending Schema

Usually you only need to specify custom Mnesia indexes and tables tuning. Riak and KAI backends don’t need it. Group your table into table packages represented as modules with handle_notice API.

-module(kvs_feed). -inclue_lib("kvs/include/kvs.hrl"). metainfo() -> #schema{name=kvs,tables=[ #table{ name = feed, container = true, fields = record_info(fields,feed)}, #table{ name = entry, container = feed, fields = record_info(fields,entry), keys = [feed_id,entry_id,from] }, #table{ name = comment, container = feed, fields = record_info(fields,comment), keys = [entry_id,author_id] } ]}.

And plug it into schema sys.config:

{kvs, {schema,[kvs_user,kvs_acl,kvs_feed,kvs_subscription]}},

After run you can create schema on local node with:

1> kvs:join().

It will create your custom schema.

KVS API

Service

System functions for start and stop service:

-spec start() -> ok | {error,any()}. -spec stop() -> stopped.

Schema Change

This API allows you to create, initialize and destroy the database schema. Depending on database the format and/or feature set may differ. join/1 function is used to initialize database, replicated from remote node along with its schema.

-spec destroy() -> ok. -spec join() -> ok | {error,any()}. -spec join(string()) -> [{atom(),any()}]. -spec init(atom(), atom()) -> list(#table{}).

Meta Info

This API allows you to build forms from table metainfo. You can also use this API for metainfo introspection.

-spec modules() -> list(atom()). -spec containers() -> list(tuple(atom(),list(atom()))). -spec tables() -> list(#table{}). -spec table(atom()) -> #table{}. -spec version() -> {version,string()}.

Chain Ops

This API allows you to modify the data, chained lists. You can use create/1 to create the container. You can add and remove nodes from lists.

-spec create(atom()) -> integer(). -spec remove(tuple()) -> ok | {error,any()}. -spec remove(atom(), any()) -> ok | {error,any()}. -spec add(tuple()) -> {ok,tuple()} | {error,exist} | {error,no_container}.

Raw Ops

These functions will patch the Erlang record inside database.

-spec put(tuple()) -> ok | {error,any()}. -spec delete(atom(), any()) -> ok | {error,any()}.

Read Ops

Allows you to read the Value by Key and list records with given secondary indexes. get/3 API is used to specify default value.

-spec index(atom(), any(), any()) -> list(tuple()). -spec get(atom(),any(), any()) -> {ok,any()}. -spec get(atom(), any()) -> {ok,any()} | {error,duplicated} | {error,not_found}.

Import/Export

You can use this API to store all database in a single file when it is possible. It’s ok for development but not very good for production API.

-spec load_db(string()) -> list(ok | {error,any()}). -spec save_db(string()) -> ok | {error,any()}.

 




Events | Privacy Policy | Feedback | Brandbook
Copyright © 2005–2016 Synrc Research Center s.r.o.