Msgcore — the structured message store

One heap.
Every field addressed.
Nothing to serialise.

Msgcore is a hierarchical store for structured data, addressable by offset or by field name — like files in a micro file system. No serialisation pass required. All data elements live in one contiguous memory image: send it over the network, save it to a file or to S3. Msgcore has been doing this since 2000.

The difference from a serialised format is what happens after the write. A serialiser emits a finished byte stream: adding a field means changing the schema, regenerating the writer and rewriting the message. Here the image is the live structure, so a store already on disk takes new fields at any point, vectors grow when you insert past their end, and the heap reallocates its base image to fit. And what will not fit need not be resident: a manager can hand a subtree’s residency to its host, which pages it in before it is read and out when it is done — so a store can describe far more data than it holds.

Watch the heap grow → Component reference Libraries & downloads

One heap · offset addressing · the same image on disk and on the wire

2000Store in service since

0Serialisation passes to save

282Flat C entry points

1Heap per store, one image

Just a few lines, and the store is on disk

The whole idea

No schema to declare, no generated accessors, no serialiser to call. A handful of lines build a two-level store and put it on disk; two more read it back. Field names travel inside the image, so the reader needs nothing but the file. The first two tabs run the same program twice — once addressing nodes by function, once by path — the third shows what they leave behind, and the last two carry that same store through MsgcoreUtils — built back from a JSON document, and built back from XML, which is the dialect that keeps the stored width.

#include "P2PmsgMgr.h"

P2PmsgMgr mgr;                                            // 1
mgr.r_name() = L"Settings";                               // 2
P3PmsgField& win = mgr.DeclareItem ( L"window",
                                   P3PmsgData ( L"" ) );  // 3
win.DeclareItem ( L"x", P3PmsgData ( (int)1240 ) );       // 4
win.DeclareItem ( L"y", P3PmsgData ( (int)820 ) );        // 5
mgr.Save ( L"settings.p2p" );                             // 6

P2PmsgMgr load ( L"settings.p2p" );                       // 7
int nX = load.SelectItem ( L"window" )                    // 8  step,
            .SelectItem ( L"x" ).c_int();                 //    then step  -> 1240

#include "P2PmsgMgr.h"

P2PmsgMgr mgr;                                       // 1
mgr.r_name() = L"Settings";                          // 2
mgr.DeclareItem ( L"window", P3PmsgData ( L"" ) );   // 3
P3PmsgField win ( mgr.RootPath2Object
                      ( L".Settings.window" ) );     //    reached by path
win.DeclareItem ( L"x", P3PmsgData ( (int)1240 ) );  // 4
win.DeclareItem ( L"y", P3PmsgData ( (int)820 ) );   // 5
mgr.Save ( L"settings.p2p" );                        // 6

P2PmsgMgr load ( L"settings.p2p" );                  // 7
P3PmsgField x ( load.RootPath2Object                 // 8  one string,
                    ( L".Settings.window.x" ) );     //    whole descent
int nX = x.c_int();                                  //    -> 1240

// A path SELECTS an existing node; DeclareItem is what creates one, which
// is why lines 3-5 are unchanged. A missing dotted descendant throws
// "Path to object does not exist" rather than answering empty.

Four nodes, one heap · each reachable by call or by path

// The same store as a JSON document. MsgcoreUtils writes it with
// oMgr >> oJson and reads it back with oMgr << oJson -- the arrow
// always points at the thing being written.

// A plain field renders as its value; a field that also carries children
// renders as an object, with its own value under ".value". An item name
// can hold neither '.' nor '@', so the sentinel keys cannot collide with
// a descendant -- that is a property of the store, not of the renderer.

// The constructor seeds the renderer with the DOCUMENT, the way SetText
// does. It is explicit, because Load() takes a FILENAME and both take an
// LPCTSTR -- an implicit one would parse a path as JSON.

#include "MsgcoreUtils/PrintJson.h"

PrintJson oJson ( LR"({
  "Settings": {
    "window": {
      ".value": "",
      "x": 1240,
      "y": 820
    }
  }
})" );

P2PmsgMgr mgr;
mgr << oJson;                         // value, attributes and descendants are
                                      // replaced -- the NAME is not. "Settings"
                                      // names the DOCUMENT, not the node, so the
                                      // wrapper key is dropped and the store keeps
                                      // the root name it already had. Address it
                                      // relatively, as below, or name the store
                                      // yourself if you want rooted paths.
if ( *oJson.GetError() )
    printf ( "line %d: %s\n", oJson.GetErrorLine(), oJson.GetError() );

P3PmsgField x ( mgr.Path2Object ( L"window.x" ) );
x.c_int64 ();                          // 1240 -- int64, NOT int32. JSON has one
                                      // number type, so c_int() would throw.

// Nothing is written until the whole document has parsed: a failure
// leaves the target exactly as it was and says where it stopped.

// The same store as XML. MsgcoreUtils writes it with oMgr >> oXML and
// reads it back with oMgr << oXML -- the arrow always points at the
// thing being written. The node's own value rides in <p2p:value>, the
// sentinel the JSON tab spells ".value".

// This is the dialect that keeps the WIDTH. p2p:type travels on every
// cell that has one, so an int32 comes back an int32 -- where JSON has
// one number type and hands back an int64. SetTypes(false) drops the
// attributes for a foreign consumer and buys back exactly that bargain.

// An element name is folded to what XML allows, so a name with a space
// in it travels in p2p:name beside it. The real name is IN the
// document rather than guessed at from the element, which is what
// makes the read direction exact instead of a best effort.

#include "MsgcoreUtils/PrintXML.h"

PrintXML oXML ( LR"(<?xml version="1.0" encoding="UTF-8"?>
<Settings xmlns:p2p="urn:ivyware:msgcore:p2p">
  <window>
    <p2p:value p2p:type="WSTR16"></p2p:value>
    <x p2p:type="int32">1240</x>
    <y p2p:type="int32">820</y>
  </window>
</Settings>)" );

P2PmsgMgr mgr;
mgr << oXML;                          // value, attributes and descendants are
                                      // replaced -- the NAME is not. <Settings>
                                      // names the DOCUMENT, not the node, so the
                                      // store keeps the root name it already had.
if ( *oXML.GetError() )
    printf ( "line %d: %s\n", oXML.GetErrorLine(), oXML.GetError() );

P3PmsgField x ( mgr.Path2Object ( L"window.x" ) );
x.c_int ();                           // 1240 -- int32, as it was stored.
                                      // The declaration is optional on the
                                      // way in: a reader that believed the
                                      // flag would refuse valid documents.

Line by line
#	Command	What it does
1	`P2PmsgMgr mgr;`	Constructs the manager: one heap, one root field, nothing on disk yet. The `(uAddrNN, nSizeInitial, nSizeMax)` overload picks the addressing width and the growth bounds instead — `VBLock_Addr64` for anything that may get large, `Addr16` / `Addr32` to buy compactness at a size ceiling.
2	`mgr.r_name() = L"Settings";`	Names the root field. Every `r_` accessor hands back a reference, so one call both reads and writes. This name is the first component of every root path into the store.
3	`mgr.DeclareItem ( L"window", P3PmsgData ( L"" ) )`	Adds a child and returns a reference to it. The descendant collection is created on demand, so there is no separate “make a node” step and no declared shape to keep in sync. A third argument of `TRUE` re-declares in place instead of adding a second field under the same name.
4–5	`win.DeclareItem ( L"x", P3PmsgData ( (int)1240 ) )`	Two typed leaves under `window`. `P3PmsgData` takes its type tag from the C++ type of the argument, and every `c_` accessor checks that tag: `c_int()` on a `double` cell throws, it does not silently truncate.
6	`mgr.Save ( L"settings.p2p" )`	Writes the heap. It is already contiguous and already self-describing, so this is a copy of bytes — no serialisation pass, no pointer fix-up. It goes to a temporary file and renames over the target, so a reader never sees a half-written store. `Save()` with no filename saves over the file the store came from; `Save ( path, true )` defragments first.
7	`P2PmsgMgr load ( L"settings.p2p" );`	The constructor-from-filename is `Load()` — shared, header-validated. The image comes back as the bytes it was: `load.Sizeof()` equals what was saved.
8	`SelectItem ( … )` vs `RootPath2Object ( … )`	The one line the two tabs disagree on. `SelectItem` steps one level and hands back a reference into the tree, so a write through it lands in the tree. `RootPath2Object` takes the whole descent as one rooted string; `Path2Object` is the same thing relative to the manager, so `L"window.x"` reaches the same leaf. All of them end at one heap offset — `GetP2Pos()` on any of them answers the same `P2Pos`. Two things the parameter names hide: `RootPath2Object` requires the leading `.` and the root’s own name, while `Path2Object` takes the path relative to the manager; and of the three calls whose argument is spelt `…Name`, `SelectObject` and `Exists` take a whole path — only `SelectItem` takes a bare name.

Attributes and the stack

Two side-cars on every field

Besides its value and its children, a field carries two more collections, each with its own path delimiter — @ for attributes and ^ for the stack, alongside the . that descends into children. All three are declared together in P2Pmsg.h, and all three hold fields, so all three nest. The store is name-addressed — no node carries an id — so a path is how a node is named in text: in a log line, a config key, a FUSE filename, a COM argument.

Attributes — `@`

Metadata that must not turn up when something walks the content tree: a unit of measurement, a sensor id, provenance, a permission. r_Attr() is a second child collection on the same field, addressed with @ instead of ., so a scan for named children is never polluted by it. An attribute is an ordinary field, so it carries a typed value, its own children and its own attributes. TreeFs surfaces this collection as .attr/ and as POSIX xattrs, precisely because it is out-of-band from the data.

P3PmsgField temp ( L"Temperature", P3PmsgData ( (double)21.5 ) );
temp.IsAttributed ();                           // false - no collection yet

temp.r_Attr ( P3PmsgField::AttrCMD_Create )
     += P3PmsgField ( L"Unit",   P3PmsgData ( L"Celsius" ) );
temp.r_Attr ()
     += P3PmsgField ( L"Sensor", P3PmsgData ( (int)7 ) );

temp.IsAttributed ();                           // true
temp.r_Attr().GetCount ();                      // 2
temp.r_Attr().Exists     ( L"Unit" );           // true
temp.r_Attr().SelectItem ( L"Unit" ).c_wstr();  // L"Celsius"

// The value is untouched, and the two collections are independent.
temp.c_double ();                               // 21.5
temp.DeclareItem ( L"Reading", P3PmsgData ( (double)21.5 ) );
temp.r_Desc().GetCount ();                      // 1
temp.r_Attr().GetCount ();                      // still 2 - separate trees

// (a) Declare the collection, then add to it. AttrCMD_Create is the house
//     style rather than a hard requirement - PushBack and DeclareItem both
//     create the block if it is absent.
temp.r_Attr ( P3PmsgField::AttrCMD_Create );
temp.r_Attr() += P3PmsgField ( L"Unit", P3PmsgData ( L"Celsius" ) );

// (b) DeclareItem on the collection - set-or-update in one call.
temp.r_Attr ( P3PmsgField::AttrCMD_Create )
     .DeclareItem ( L"Unit", P3PmsgData ( L"Celsius" ),  /*bUpdate*/ true );

// (c) Build the attribute as the field it is. A unit of measurement is a
//     good example of one that wants structure of its own.
P3PmsgField unit ( L"Unit", P3PmsgData ( L"Celsius" ) );
unit.DeclareItem ( L"Symbol", P3PmsgData ( L"\u00B0C" ) );
unit.DeclareItem ( L"SI",     P3PmsgData ( L"kelvin" ) );
unit.DeclareItem ( L"Offset", P3PmsgData ( (double)273.15 ) );
temp.r_Attr ( P3PmsgField::AttrCMD_Create ) += unit;     // deep copy

// (d) Hold the collection once, then work through it. Writes through a
//     reference land in the store.
P3PmsgAttr& attr = temp.r_Attr ();
attr += P3PmsgField ( L"Sensor", P3PmsgData ( (int)7 ) );
attr.SelectItem ( L"Sensor" ).c_int ( 9 );               // write, not read
attr.Delete ( L"Sensor" );
attr.Truncate ();                                        // empty the collection

P2PmsgMgr mgr;
mgr.r_name() = L"Store";
mgr.r_Desc() += temp;                                            // Temperature now lives in the store

mgr.RootPath2Object ( L".Store.Temperature"         );           // the field    -> 21.5
mgr.RootPath2Object ( L".Store.Temperature@Unit"    );           // the attribute -> "Celsius"
mgr.RootPath2Object ( L".Store.Temperature@Sensor"  );           // -> 7
mgr.RootPath2Object ( L".Store.Temperature.Reading" );           // a DESCENDANT, not an attribute

// A bare delimiter names the collection itself.
mgr.RootPath2Object ( L".Store.Temperature@" ).IsAttr ();        // true
mgr.RootPath2Object ( L".Store.Temperature." ).IsDesc ();        // true

// Relative to the manager, without naming the root:
mgr.Path2Object ( L"Temperature@Unit" );

// A miss on an '@' component answers void. A miss on a dotted descendant
// THROWS - "Path to object does not exist" - and so does a root path that
// does not start at the root.
mgr.RootPath2Object ( L".Store.Temperature@Nobody" ).IsVoid ();  // true

// Offset and path are the same address in two notations, and convert:
P3PmsgField& live = mgr.SelectItem ( L"Temperature" );
P2Pos   pos  = live.GetP2Pos ();                                 // an offset into the heap
CString path = live.GetPath   ();                                // the same node, as a string
mgr.P2Pos2Path ( pos ) == path;                                  // true

The stack — `^`

Every field carries its own stack. Push() copies the whole item — name, data, attributes and descendants — into a fresh block linked off the item; Pop() copies it back and drops the copy. Pushes nest. It is a scoped override: snapshot, mutate freely, then unwind, without holding a separate copy of your own. The snapshot keeps the name it was pushed under, so the name will not tell it from the live item — the handle will.

P3PmsgField quote ( L"Quote", DataBSTR08 ( L"100.25" ) );
quote.IsStacked ();                       // false

quote.r_Stck().Push ();                   // snapshot the WHOLE item
quote.IsStacked ();                       // true

quote = P3PmsgName ( L"Quote-Revised" );  // rename in place; the cell survives

if ( quote.IsStacked() )
    quote.r_Stck().Pop ();                // the override unwinds
quote == L"Quote";                        // true - and the copy is dropped

// (a) Pushes nest, and unwind last-in-first-out.
quote.r_Stck().Push ();
quote.r_Stck().Push ();
quote.r_Stck().Pop  ();                // back to the second snapshot
quote.r_Stck().Pop  ();                // back to the first
P3Pmsg_GetStckDepth ( &quote, 0 );     // how deep it is now

// (b) A snapshot is a whole item, not just its value: children and
//     attributes present at the push are inside it, later ones are not.
P3PmsgItem q ( L"Quote" );
q.r_Desc ( P3PmsgField::AttrCMD_Create );
q.r_Desc() += P3PmsgField ( L"Bid" );
q.r_Attr ( P3PmsgField::AttrCMD_Create ) += P3PmsgField ( L"Currency" );
q.r_Stck().Push ();                    // Bid and Currency go in with it
q.r_Desc() += P3PmsgField ( L"Ask" );  // added AFTER - live only

// (c) Read a pushed value back without popping - the stack keeps one
//     accessor per type, and each throws if asked for the wrong one.
q.r_Stck().r_name ();                  // the pushed NAME
q.r_Stck().r_data ();                  // the pushed VALUE
q.r_Stck().r_item ();                  // the pushed ITEM

// (d) += deep-copies into a collection, so a child you inserted is not the
//     live one. Fetch the live handle back out before pushing it.
mgr.r_Desc() += P3PmsgField ( L"BHP" );
P3PmsgField live = mgr.RootPath2Object ( L".Store.BHP" );
live.r_Stck().Push ();

// The item built in (b), addressed by path instead of by call.
P3Pmsg_SelectObject ( &q.r_Object(), L"^"          );                // the snapshot - name 'Quote'
P3Pmsg_SelectObject ( &q.r_Object(), L"^.Bid"      );                // Bid, as it stood
P3Pmsg_SelectObject ( &q.r_Object(), L"^Bid"       );                // same - the '.' is optional
P3Pmsg_SelectObject ( &q.r_Object(), L"^.Ask"      );                // void - added after the push
P3Pmsg_SelectObject ( &q.r_Object(), L"Ask"        );                // Ask - the live item has it
P3Pmsg_SelectObject ( &q.r_Object(), L"^^"         );                // the push before that
P3Pmsg_SelectObject ( &q.r_Object(), L"^@Currency" );                // == "@^Currency"

// The same shapes as a full root path. In this tree .Store.BHP was pushed
// after gaining a child 'Last', and .Store.RIO was never pushed at all.
mgr.RootPath2Object ( L".Store.BHP"          );                      // the live item
mgr.RootPath2Object ( L".Store.BHP@Currency" );                      // its attribute
mgr.RootPath2Object ( L".Store.BHP^"         );                      // its snapshot - a DIFFERENT object
mgr.RootPath2Object ( L".Store.BHP^.Last"    );                      // a child of the snapshot
mgr.RootPath2Object ( L".Store.RIO^"         );                      // void - RIO was never pushed

// Only the handle tells a snapshot from the live item: the two answer the
// same name, and IsVoid() is false for both.
P3PmsgField ( mgr.RootPath2Object ( L".Store.BHP"  ) ).GetP2Pos ();  // 259
P3PmsgField ( mgr.RootPath2Object ( L".Store.BHP^" ) ).GetP2Pos ();  // 1009

The path grammar
Path	Resolves to
`Item`	The child `Item` of the item you are standing on.
`Item.Child`	A descendant, one level down.
`Item@Tag`	The attribute `Tag`.
`Item/Child` · `Item\Child`	A descendant. `/` and `\` are accepted on input as synonyms for `.` — convenient when a path arrives from a file system.
`Item@` · `Item.`	The attribute collection, and the descendant collection, as objects in their own right.
`Item^`	`Item` as it stood before its last push.
`Item^^`	Before the push before that. One `^` too many answers void; it does not fail.
`Item^.Child` · `Item^Child`	A descendant of the snapshot. The `.` after a `^` is optional, as it is after a name.
`Item@Tag^`	A pushed attribute — attributes are items and carry stacks too.
`Item@^Tag` · `Item^@Tag`	The same object. `^` commutes with `@` and with `.`, because a snapshot is a whole item rather than a fragment of one.
`List@Tag` · `Vect@Tag`	A list and a vector are items, so they carry their own attributes and children.
`.Root.Item^`	Any of the above as a rooted path through `RootPath2Object`. `Path2Object` takes the same path relative to the manager.

Paging

Large datasets held outside the heap

A store and its data are normally the same thing: the tree is in the heap, the heap is the image, and the image is what you save. Paging breaks that identity for a chosen subtree. The tree still describes it — the nodes, the names, the shape are all there — but the bytes can live somewhere else entirely and arrive only when something reads them. Msgcore_c.h puts it in one line: a manager can delegate the residency of a subtree to its host, so a store may describe far more data than it holds.

Residency, delegated

You install two callbacks. When a paged dataset is about to be read, the core calls your page-in with the P2Pos of the item; you fetch it from wherever it really lives — a disk, a database, a socket — and answer 1 for resident or 0 for could not. When the work is finished it calls page-out, with a flag saying whether your copy needs writing back first. SafeDSetPaging is that same pair expressed as a scope, carrying the summary state built from PAGESUMM_DIRTY, CACHED, FLUSH, NOMERGE and DECACHE.

What it buys is that paths still resolve. The structure is addressable whether or not the bytes are home, so a consumer can walk it, search it and hand out addresses for nodes whose contents have never been fetched — and the fetch happens underneath, at the moment of the read, rather than being something the caller has to orchestrate.

#include "P2PmsgMgr.h"

// The host answers with the data. posItem names the node being asked
// about; nKey is the registration key the manager hands back to you.
BOOL CALLBACK PageIn  ( PINT_PTR nKey, P2Pos posItem );
BOOL CALLBACK PageOut ( PINT_PTR nKey, P2Pos posItem, BOOL bFlush );

P2PmsgMgr mgr ( L"catalogue.p2p" );
mgr.PageRegistration ( nKey, PageIn, PageOut );      // install the pair

mgr.PageDatasetIn  ( posRows );                    // fetch it now
mgr.PageDatasetOut ( posRows, /*bFlush*/ TRUE );   // write back, then drop

// Or let scope do it: in on construction, out on destruction.
{
    SafeDSetPaging oRows ( mgr, oRowsItem );        // resident from here
    oRows->r_Desc().GetCount ();                     // read it freely
}                                                    // ... to here

// Re-count after adding or removing elements, so the summary is honest.
mgr.PageSumm ( oRowsItem, /*additions*/ 12, /*removals*/ 0 );

#include "Msgcore_c.h"

// The same two callbacks, without the CALLBACK convention and without
// the key: the pUser you install is handed back to you unchanged.
static int PageIn  ( void* pUser, unsigned long long p2pos );
static int PageOut ( void* pUser, unsigned long long p2pos, int bFlush );

MsgMgrHandle hMgr = msgcore_mgr_open_file ( L"catalogue.p2p" );
msgcore_mgr_set_paging_sinks ( hMgr, PageIn, PageOut, pUser );

msgcore_mgr_page_dataset_in  ( hMgr, p2posRows );
msgcore_mgr_page_dataset_out ( hMgr, p2posRows, /*bFlush*/ 1 );

// 1 means "done, the data is resident"; 0 means "could not". With no
// sink installed both calls are successful no-ops, so paging costs
// nothing at all until you opt into it.

// The populate sink is the third one: it fills a subtree, rather than
// moving a whole dataset in and back out again.
msgcore_mgr_set_populate_sink ( hMgr, Populate, pUser );

// A Save walks the whole tree. With sinks installed, that walk would
// page the ENTIRE virtual dataset in just to write it out -- so the
// walk runs with the registration saved and cleared.
mgr.PageRegistrationPush ();      // save the sinks, clear them
mgr.Save ( L"catalogue.p2p" );    // nothing pages in
mgr.PageRegistrationPop  ();      // put them back

// The RAII form, which is what you want if anything between can throw.
{
    SafeRegistrationPush oQuiet ( &mgr );
    mgr.Save ( L"catalogue.p2p" );
}

// PUSH DOES NOT NEST. Pushing twice without an intervening pop asserts
// in a debug core, and loses the first saved set in a release one.

The contract — synchronous, and the core waits

This is not a background fetch. Msgcore_c.h is explicit about it: the core calls the sinks synchronously, on the thread performing the access, and waits for the answer — a page-in that has not returned yet is data that is not there, so there is no version of this that defers. A slow host is a stalled read, not a fetch in flight.

Two rules follow. A sink must not block for long, and it must not re-enter the manager that called it beyond the subtree it was asked to populate. The trigger sink’s contract is the opposite on every one of these points, and the header says in as many words not to model one on the other.

The paging surface
C++	Flat C	What it does
`PageRegistration ( key, in, out )`	`msgcore_mgr_set_paging_sinks`	Install the page-in / page-out pair. Passing null for both clears the registration and releases the record.
`PageRegistration ( key, populate )`	`msgcore_mgr_set_populate_sink`	Install the populate sink, which fills a subtree rather than moving a whole dataset.
`PageDatasetIn ( pos )`	`msgcore_mgr_page_dataset_in`	Ask the host for the dataset rooted at that node, now. A successful no-op when no sink is installed.
`PageDatasetOut ( pos, bFlush )`	`msgcore_mgr_page_dataset_out`	Hand it back — written out first if `bFlush`, simply dropped if not.
`PageSumm ( item, adds, removes )`	`msgcore_mgr_page_summ`	Re-count a paged item after a mutation; answers the new summary count.
`PageRegistrationPush ()`	`msgcore_mgr_page_registration_push`	Save the installed sinks and clear them, so a section of code runs with paging suppressed. Does not nest.
`PageRegistrationPop ()`	`msgcore_mgr_page_registration_pop`	Restore them.
`SafeDSetPaging`	—	The in/out pair as a scope: paged in by the constructor, out by the destructor, summary state carried between.
`SafeRegistrationPush`	—	Push and pop as a scope, for a suppressed section that can throw.

Watch the heap grow

Five fields go into one heap. The fifth does not fit, so the heap reallocates its base and copies — and every offset in the image is still correct, because none of them ever named an address. Then the whole thing is written to disk as the bytes it already is.

Caller code

Inside the manager

What the store gives you

Design properties

Addressing

Offsets, not pointers

Every reference inside the store is a displacement from the base of the heap. Growth reallocates that base — and every reference in the image is still correct, because none of them named an address in the first place.

Persistence

Save is a copy

The store is already contiguous and already self-describing, so writing it is writing bytes. Save goes to a temporary file and renames over the target, so a reader never sees a half-written store.

Shape

Fields, children, attributes

A field carries a name, a typed value, an ordered set of children and a set of @-qualified attributes. Native types, strings, blobs, XML and images all live in the one structure, addressed by field and by path.

Containers

Lists, vectors, stacks, cursors

The container types are not adapters over the store; they are laid out inside its heap. A cursor survives mutation of the ring it is scanning, which is what makes by-name traversal of a live store workable.

Portability

One image, two widths

An endian sentinel travels with every image and doubles as its layout generation, so a store written under one byte order or addressing width is recognised — rather than silently misread — by another.

Reach

C++, C, Java, COM

A C++ class API for code built with the same toolset, and a flat extern "C" ABI of 282 entry points for everyone else — C, .NET through P/Invoke, FUSE front ends. For Java all 282 are bound through java.lang.foreign, and a coverage test fails the build if the generated layer ever falls behind Msgcore_c.h. MsgFacade wraps the object model without macros, and MsgcoreCom puts a store one CreateObject away from PowerShell, VBScript and C#. See the facade page.

Diagnostics

One event system

P2Pevent carries errors and diagnostics as throwable C++ exceptions, or as data attached to a store — the same object TargetCore routes across a network.

Licence

Apache 2.0

Msgcore is licensed under the Apache 2.0 licence, alongside TargetCore. The integration libraries and reference applications are already free downloads with full source.

Scope

What it is not

The store is not internally synchronised — one store is serialised by its caller — and it has not been reviewed as a network-facing parser. Treat an image from a source you do not control as unsafe to load.

Path resolution is not on the flat C ABI. Msgcore_c.h carries the emitter, msgcore_mgr_p2pos2path, and no resolver — so a C, Panama or .NET caller can ask a node for its path, but addresses it back by P2Pos or by name. Path2Object and RootPath2Object are on the C++ classes, which are toolchain-pinned rather than a supported 1.x surface.

Two libraries, one lineage

How they fit

Msgcore and TargetCore are separate products with a one-way dependency. Msgcore knows nothing about networks; TargetCore does not define a payload format.

MsgcoreDefines the store, the fields and the P2Piomage frame. In service since 2000.

TargetCoreAddresses, encrypts and moves that frame between hubs. In service since 2002.

The ruleMsgcore does not depend on TargetCore. The layering is acyclic, and always has been.

In practiceThe message you build with Msgcore is the message a hub routes. There is no conversion step.

The type reference → TargetCore walkthrough

The pieces

Type model

Seven families carry the whole store. The full reference — the containers, the flat C ABI and the two rules a caller has to honour — is on the architecture page.

Msgcore type families
Type	Responsibility
`P2PmsgMgr`	The store manager. Owns the heap, the root field, the save and load paths, path lookup and triggers.
`P3PmsgField`	A named node: a value, an ordered set of children, and a set of attributes. The unit everything else is built from.
`P3PmsgData`	The typed value a field holds — native types, strings, BSTRs, blobs, XML and images.
`MsgVBHeap`	The offset-addressed heap and its block allocator: growth, the free list, and the addressing widths.
`MsgList` · `MsgVect` · `MsgStck` · `MsgCurs`	The container types over a field's children, laid out inside the heap rather than over it.
`P3PmsgBSTR` · `P2Piomage`	String storage, and the packed sync-header frame that leaves the process — the shape TargetCore encrypts and moves.
`P2Pevent`	Events and exceptions. Throwable, attachable to a store, and shared with TargetCore.

Full architecture → Downloads

Built on it

In production

Library

TargetCore

The routing kernel. Every message a hub queues, encrypts and dispatches is a Msgcore image; the kernel adds the address, the wire and the handler map.

See the walkthrough →

Application

Chartboard

Workspaces, chart stacks, portfolios and scanner results are stores — which is why a workspace file loads without a parse and a chart survives a round trip intact.

Explore Chartboard →

Reference server

AKAserva

An HTTP and WebSocket server where a parsed request is a field tree, decorated by handlers and translated back into a response.

One heap.Every field addressed.Nothing to serialise.