Skip to content
Docs
Daemons
Daemons Introduction

Daemons Introduction

The Daemons (or Imps, as you want) are GoRoutines responsible for fetching, handling, and storing data from various sources to make them available to the rest of the system.

When working with APIs, we usually have to deal with a Database, with external endpoints etc. Querying all of these, especially the Blockchain environment can be really slow, which is not compatible with a proper API.

To reduce the loading time and ensure we always have the most recent data, we need to have a way to fetch the data from the Blockchain, store it in a local shared database, and then expose it to our application: this is the role of the Daemons.

💡

Goroutines are functions or methods that run concurrently with other functions or methods. Goroutines can be thought of as lightweight threads. The cost of creating a Goroutine is tiny when compared to a thread. Hence it's common for Go applications to have thousands of Goroutines running concurrently. Learn more.

What is a Daemon doing?

Every Daemon is responsible for a specific part of the application but is built on top of a common base. We are therefore using the same arguments and the calling functions are almost the same.

Arguments

The arguments are always the same:

  • chainID: the chain id of the Blockchain we are working with. This will be used to get the data for the correct working environment but also to same them correctly, avoiding mixing data from different chains.
  • WaitGroup: a sync.WaitGroup that will be used to sync a group of daemons together, so we can ensure they all finish before we continue with project initialization.

Process

func runDaemon(chainID uint64, wg *sync.WaitGroup, delay time.Duration, performAction func(chainID uint64)) {
	isDone := false
	for {
		performAction(chainID)
		if !isDone {
			isDone = true
			wg.Done()
		}
		time.Sleep(delay)
	}
}

The process is the same:

  1. We first try to fetch the data from the local database. Having these ensure we always have at least something to serve, even if some endpoints are down.
  2. Once the data from the database is recovered, we init an infinite loop that will perform its task every interval time.
  3. The first action of the loop will be to perform its task, no matter which one it is.
  4. Once the task is done (task can soft fail but the loop will continue), the sync.WaitGroup will be notified and the Done event will be triggered. This will indicate that the group is ready to proceed (first run is done).
  5. The Daemon will sleep for interval time and then repeat the process from step 3.

Database and cache

Cache

The cache (or store) is the in-memory data we use to get fast access to the data we need. It's not a database or any kind of system, just a global variable declared and available in which we can set and get the data we need.

var TokenList = make(map[uint64][]common.Address)

// set
TokenList[chainID] = myUpdatedTokenList

// get
myTokenList = TokenList[myChainID]

Database

The Daemon will use a local database to store the data it fetches. This database will be stored in the ystore folder. We are using the Badger system for that, which is a basic Key-Value Store.

In a nutshell, it's fast and easy to use. It will create some files in the ystore directory to save the data in it. This system ensure we always have the latest valid data available in all circumstances, avoiding data loss or endpoints being down because one of the data sources is down. In this situation data may obviously be outdated, but the latest valid one will be served.

Key to the database is the chainID of the chain we are working with, appended to some specific key. We have, for instance, VaultsFromMeta1 for the VaultsFromMeta data on chain 1. The value is an interface, aka an abstract object we are required to correctly decode to be accessed.

Save the data

func SaveInDBForChainID(key string, chainID uint64, value interface{}) {
	SaveInDB(key+strconv.FormatUint(chainID, 10), value)
}

[...]

store.SaveInDBForChainID(`myDataKey`, myChainID, myData)

Access the data

func LoadFromDBForChainID(key string, chainID uint64, dest interface{}) error {
	err := LoadFromDB(key+strconv.FormatUint(chainID, 10), &dest)
	if err != nil {
		return err
	}
	return nil
}

[...]

temp := myDataType
store.LoadFromDBForChainID(`myDataKey`, myChainID, &temp)
myCachedData[myChainID] = temp
Introduction
List of Daemons