The DDC consists of two kind on nodes: CDN and Storage. Both kinds are daemon applications and communicate with each other with HTTP requests and data wrapped into protocol buffers. Both kinds share some code:

  1. SDK-GO contains different shared code: CID builder, crypto primitives and topology helpers,
  2. DDC-Schemas contains protocol buffer layouts of the types used in the DDC.

The DDC consists of two kind on nodes: CDN and Storage. Both kinds are daemon applications and communicate with each other with HTTP requests and data wrapped into protocol buffers. Both kinds share some code:

  1. SDK-GO contains different shared code: CID builder, crypto primitives and topology helpers,
  2. DDC-Schemas contains protocol buffer layouts of the types used in the DDC.

SDK and pieces

Also there is a SDK-JS module in DDC. This is the code working on the client side and making requests to the CDN cluster. The important notion here is a piece. A piece is an abstraction that represents a unit of data stored in the DDC. It doesn't have a fixed size and can represent fully logically completed data or part of it. It consists of:

  1. data - raw bytes that this piece contains,
  2. bucketId - bucket (in smart contract sense) that this piece belongs to,
  3. Array of tags - key/value pairs that contain metadata attached to the piece, for example, encryption options or file name,
  4. Array of links - If this piece is interpreted as a file, the linked pieces make up the file content, otherwise empty.

Datastore and buckets

Nodes store data locally in badgerDb database. The database API is wrapped into datastore package that actually adds a notion of buckets. Datastore bucket (not to be confused with the buckets in smart contract) is like a partition, a subset of keys that store similar kinds of value. For example, currently, Storage nodes have 4 buckets:

  1. Piece bucket - contains stored pieces data
  2. Token bucket - contains CIDs of the stores pieces
  3. Tag bucket - contains indexes - CIDs of the pieces that has particular tag
  4. Metrics bucket - contains statistics counter like the number of stored bytes in every bucket

CRUD (Read/Write/Update)

Read

  1. SDK provides reading either pieces or files. For this ddcClient.read(ddcUri, readOptions) API is used [code].
    1. it reads the piece requested [code],
      1. requests CDN node with /api/v1/rest/pieces/<cid>?bucketId=<bucket> GET HTTP request and signs it [code],
      2. verifies that the received piece conforms to its cid [code],
      3. emits ACK [code]
      4. ⚠️ no response signature verification
    2. decrypts it if requested [code],
    3. if IFILE-type ddcUrlwas requested, the piece links are used to read the file parts and then are concatenated [code],
  2. CDN nodes handle the request with pieceRouterV1.getPiece [code].
    1. Authenticates the request by verifying its signature [code],
    2. Checks the bucket rent and access [code],
    3. Checks if the requested piece is in the cache, also updates cache statistics [code],
    4. Requests Storage nodes with HTTP GET /api/rest/pieces/<cid>?bucketId=<bucket>:
      1. collects the list of nodes that must have the piece according to the topology [code],
      2. starting from the random one, queries each node in the loop until the request is successful [code],
    5. Returns piece data and its cid, updates metric counters [code].