Redis is a powerful, data structure-based database designed to provide ultra-fast responses to user queries. It is referred to as an in-memory database, meaning that it stores its entire dataset in memory, making access times extremely fast. However, Redis also offers mechanisms to persist data to disk, allowing for durability even after a system reboot.
At the core of Redis are its rich and flexible data structures — strings, hashes, lists, sets, and sorted sets — each of which offers powerful operations for manipulating data in memory. In this article, I’ll explore how a simplified version of Redis works behind the scenes, focusing on basic concepts like in-memory storage, data persistence, and simple TCP communication.
Let’s get started!
Client Protocol and Network Communication:
Redis communicates with clients using the Redis Serialization Protocol (RESP), a simple, binary-safe protocol that is efficient for both clients and servers. It operates primarily over TCP connections, but UNIX sockets can also be used for inter-process communication on the same machine.
While RESP is crucial to Redis’s efficiency, we’ll simplify things in this implementation. Instead of using RESP, we’ll build a basic TCP server that listens on port 6379 (the default Redis port).
Lets build a TCP connection :
l, err := net.Listen("tcp", ":6379")
if err != nil {
fmt.Println(err)
return
}
fmt.Println("****starting TCP Server*****")
con, err := l.Accept()
if err != nil {
fmt.Println(err)
return
}
defer con.Close()
After building a TCP connection to port 6379, we can listen to the port using the telnet command. We’ll build basic operations like PING, SET, and GET, similar to how Redis functions.
Ping Command
Let’s begin with the simplest operation: the ping command. In Redis, the PING command returns PONG, verifying that the server is running. Here’s a basic version:
func ping() {
fmt.Println("pong")
}
This ping() function prints “pong” to the terminal, confirming the server is responsive.
Set Command
Redis uses the SET command to store a key-value pair in memory. We’ll implement a simplified version using a map data structure to store key-value pairs:
//this type is declared in the types package
type StoreType map[string]string
//making a hashmap structure using StoreType from the type package
var store = make(types.StoreType)
func set(args string) {
text := strings.TrimSpace(args)
parts := strings.SplitN(text, " ", 2)
if len(parts) != 2 {
fmt.Println("Invalid arguments for set command")
}
key := parts[0]
value := parts[1]
store[key] = value
//persist the value
storage.SaveDataToFile(store)
fmt.Println("store: ", store)
}
In this method, we parse the input and store the key-value pair in memory. Spaces split the key and value. If the input is invalid, the function will terminate early. Once stored, the method persists the key-value pair to a file for durability.
Now, the key value is saved in our memory. What if our application is stopped, and the value is lost?
Data Persistence
While Redis stores data in memory for fast access, it also offers several mechanisms for persistence, like RDB snapshots and Append-Only Files (AOF). In our implementation, we’ll simply persist key-value pairs into a file (store.txt). Here’s how we can implement file-based persistence:
var StoreFile = "store.txt"
var FileMutex = &sync.Mutex{}
var Store = make(types.StoreType)
func SaveDataToFile(store types.StoreType) {
FileMutex.Lock()
defer FileMutex.Unlock()
file, err := os.OpenFile(StoreFile, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("Error when opening file : ", err)
_, cerr := os.Create(StoreFile)
if cerr != nil {
fmt.Println("Error from creating file: ", cerr)
}
}
defer file.Close()
writer := bufio.NewWriter(file)
for key, value := range store {
_, err := fmt.Fprintf(writer, "%s=%s\n", key, value)
if err != nil {
return
}
}
This function persists key-value pairs to store.txt. We use a mutex to ensure that file writes are thread-safe. This simplified version of Redis’s AOF feature logs each write command to a file.
Get-Command
Next, we’ll implement the GET command, which retrieves the value for a given key from memory:
func get(args string) {
text := strings.TrimSpace(args)
fmt.Println("Get for key : ", text)
if text == "" {
fmt.Println("Invalid arguments for get command")
}
if value, found := storage.Get(text); found {
fmt.Println("value is: ", value)
} else {
fmt.Println("Key not found")
}
}
This function retrieves the value for the given key from memory. If the key is not found, it prints “Key not found.”
Storage Access
Our Get() method accesses the in-memory store to find the key-value pair:
func Get(key string) (string, bool) {
FileMutex.Lock()
defer FileMutex.Unlock()
value, found := Store[key]
return value, found
}
Here, we retrieve the value from the Store map, our in-memory database.
Let’s consider a situation: Do we just get the key from a store txt file where we persist our data? The answer is no. As we discussed earlier, we are trying to build an in-memory database, so our key value should be in memory.
Loading Data from File
To maintain consistency across application restarts, we need to load data from store.txt into memory when the application starts. This is akin to Redis’s RDB loading mechanism. Here’s how we load the data:
func LoadDataFromFile() {
FileMutex.Lock()
defer FileMutex.Unlock()
file, err := os.Open(StoreFile)
if err != nil {
fmt.Println("Error when opening file : ", err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := scanner.Text()
parts := strings.Split(line, "=")
if len(parts) != 2 {
fmt.Println("Skipping malformed line:", line)
continue
}
key := strings.TrimSpace(parts[0])
value := strings.TrimSpace(parts[1])
Store[key] = value
}
}
This function reads key-value pairs from store.txt and loads them into memory. This method is only called once during application startup to prevent performance bottlenecks.
storage.LoadDataFromFile()
Handling Commands
Now, let’s handle incoming commands. Our server reads input and routes it to the appropriate handler:
for {
buf := make([]byte, 1024)
storage.LoadDataFromFile()
i, err := con.Read(buf)
if err != nil {
if err == io.EOF {
break
}
fmt.Println("error from reading : ", err.Error())
os.Exit(1)
}
command := strings.TrimSpace(string(buf[:i]))
fmt.Println(command)
for k, handlers := range handlers.Handlers {
if strings.HasPrefix(command, k) {
args := strings.TrimPrefix(command, k)
handlers(args)
}
}
}
The input is read and trimmed, then routed to the correct handler based on the command prefix.
Command Handlers
We define handlers for each command in a separate package:
var Handlers = map[string]func(string){
"ping": func(args string) {
ping()
},
"set": func(args string) {
set(args)
},
"get": func(args string) {
get(args)
},
}
Each command is mapped to its respective handler, similar to how Redis parses and executes commands.
Final Output
Terminal 1: First run : go run main.go
****starting TCP Server*****
Terminal 2 : telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set key value
Terminal 1: It will give the output in application terminal:
set key value
store: map[key:value]
Terminal 2: get key
Terminal 1: It will give the output in application terminal:
get key
Get for key : key
value is: value
Conclusion
Building a simple Redis-like application helps us understand the core concepts of in-memory storage, persistence, and network communication. While this version is simplified, Redis provides more advanced features like robust persistence (RDB and AOF), atomicity, and efficient protocols like RESP.
Feel free to leave any feedback or suggestions on improving this!
