Introduction

In the previous article we created various class models that represented various parts of the data which we would want to store.

In this article, we would look at how we can actually store these models using a file storage as discussed in the previous article

Building the File Storage Engine

Now we are able to create an instance using the dictionary which is great because it keeps track of all our instances. But there is a problem: every time we restart the program, we would have to recreate everything from scratch.

To solve that, we need to find a way to save all the data so that when we restart the program, it could just be reloaded and then we can add new instances to our previous ones.

Hence, in building our data storage, we need to ensure persistency.

Persistency refers to the ability of data to remain stored and accessible even after the program or device is shut down or restarted. It ensures that information remains intact over time, allowing users to retrieve and utilize it whenever needed. Think of it like saving a document on your computer—once saved, the document stays there until you decide to delete or modify it.

How to ensure data persistency

Here is a proposed solution to that

• We would simply need to write the objects to a file and save it.

• This would mean, converting the instance to a dictionary and then writing it into a file in JSON format to be saved.

Now you might ask why we should use this approach. Well here are the reasons

1. Python doesn't know how to convert a string into a dictionary easily.

2. The dictionary isn't human readable.

3. Using the dictionary in another language might be a bit difficult since different languages have different ways of implementing dictionaries.

JSON in Simple Terms:

JSON, short for JavaScript Object Notation, is a lightweight data format used for storing and exchanging information between different systems. It resembles a structured text format, making it easy to read and write by both humans and computers. It's like packaging up data in a neat box so it can be easily shared and understood by different programs.

So, the process would look like this diagrammatically.

<class 'BaseModel'> -> to_dict() -> <class 'dict'> -> JSON dump -> <class 'str'> -> FILE -> <class 'str'> -> JSON load -> <class 'dict'> -> <class 'BaseModel'>

And more like this in a pictorial view

Read here for the difference between the dictionary and JSON

Storing the Objects

From our diagram, the steps for serialization (storage to the file) would be as follows:

1. Convert the instance to dictionary using to_dict().

2. Convert the dictionary to JSON using a method known as json.dump(s).

3. This would produce a pure string and then we save it to a FILE.

The steps for deserialization (retrieval from the file) would be as follows:

1. Open the FILE which has the JSON object.

2. Retrieve the JSON using the method json.load(s).

3. Convert this to a dictionary and use that dictionary to create an instance.

Creating the Engine Folder

We would create a separate folder to handle this entire process. This folder would be in our "models" folder.

mkdir engine

Setting up the File Storage Class

We would set up a class to handle the serialization and deserialization. This class would also have other attributes which would come in handy for our usage. In the engine folder, create two files __init__.py and file_storage.py

cd engine
touch __init__.py file_storage.py

NOTE: The presence of an "init" file makes the "engine" folder a package.

Role of the File Storage Class

Our FileStorage class would be responsible for serializing instances to JSON and deserializing JSON files to instances.

Now let's build our FileStorage Class 🔥

The FileStorage Class

To build out the class attributes and various methods we need to know what exactly we want. I created a list for that.

List of things to be done

1. We want to have an objects dictionary, which stores all the instances.

2. We need a way to save these objects in the dictionary into a file (serialize)

3. From point two, we would have to create/get the file which would serve as the storage for now.

4. We would figure out how to add a new instance object to the dictionary of objects.

5. A function would be needed to retrieve all these instances in the dictionary of objects.

6. A function to deserialize the data from the file. This would be very crucial as we want to ensure persistency. When the program relaunches, the dictionaries in the file are simply used to recreate all the instances before a user begins to interact with the system again.

From the above we would have to create the following attributes and methods

Attributes

• __fiile_path: This would be a string to store our file path (ex. "file.json").

• __objects: This would help us with the dictionary of objects spoken about earlier. We would store all the objects in this regard <classname>.id.

  Example, to store a User object with id = 321423, the key for this entry would be User.321423. We just want to have a way to access each instance by their names and id (which is unique). This would be very handy as two instances would always have different ids.

Methods

• all(self): This would return the dictionary of objects ie. __objects. (See point 5 of the lists I created).

• new(self, obj): This adds a new created instance to the dictionary of objects ie. __objects (See point 4 of the list). The key for this would be <obj classname>.id just as we saw in the attributes above.

• save(self): This method would simply save or serialize all the items in the dictionary of objects to the json file ie __file_path. This fulfils point 2.

• reload(self): As explained in point 6 of our list, we would deserialize (convert the JSON to a dictionary) the JSON file. In this function, we would that data to recreate all the instances for every session which the user launches the program. This would make our storage much persistent, as input stored is always regenerated and new inputs added are stored in the dictionary of objects and then subsequently stored in the file.

Let's start coding 🔥

Create FileStorage class and add the Attributes of the Class

In your file_storage.py file add these lines of code

#!/usr/bin/python3
"""
Module for file storage
"""
import json

class FileStorage:
    """
    - Serializes instances to JSON
    - Deserializes JSON to instances
    """

    __file_path = "file.json"
    __objects = {}

NOTE: Remember to import the JSON module as we would be using it in the file for both serialization and deserialization

All() method

    def all(self):
        """Returns dictionary of objects"""
        return self.__objects

new(self, obj) method

    def new(self, obj):
        """
        Sets an object in __objects with key <obj_class>.id
        """
        key = "{}.{}".format(obj.__class__.__name__, obj.id)
        self.__objects[key] = obj

Analysis

• This takes an object (an instance) as an argument.

• obj.__class__.__name__ : This retrieves the classname of the object.

• obj.id : This gives us the id of the object.

• Since we are simply populating the __objects dictionary, we do so with a key and value pair. Our key would be the <obj_class>.id and the value would be the entire obj which has been passed.

save(self) method

Here's a recap of the diagram for serialization

<class 'BaseModel'> -> to_dict() -> <class 'dict'> -> JSON dump -> <class 'str'> -> FILE

Now let's code the function

 def save(self):
        """
        Serializes __objects into JSON file (path __file_path)
        """
        obj_dict = {}
        for key, obj in self.all().items():
            obj_dict[key] = obj.to_dict()

        with open(self.__file_path, "w", encoding="UTF-8") as text_file:
            json.dump(obj_dict, text_file)

Analysis

• I created a temporary dictionary of objects (obj_dict) and then added every item in the original __objects to it using a for loop. You could also use the __objects directly in the "with" statement.

• Open a file (file.json) in write mode and then dump the dictionary in the file.

reload(self)

Here's a recap of the diagram for deserialization

FILE -> <class 'str'> -> JSON load -> <class 'dict'> -> <class 'BaseModel'>

Now let's code the function

def reload(self):
        """
        Deserializes the JSON file to __objects only if JSON file exists
        Otherwise do nothing. If the file doesn't exist no exception should
        be raised
        """
        from models.base_model import BaseModel
        from models.user import User
        from models.city import City
        from models.place import Place

        # Import others as needed and add to the calss map dictionary below

        class_map = {
                    'BaseModel': BaseModel,
                    'User': User,
                    'City': City,
                    'Place': Place                
            }

        try:
            with open(self.__file_path, "r", encoding="UTF-8") as text_file:
                obj_dict = json.load(text_file)

                for key, val in obj_dict.items():
                    class_name = val['__class__']
                    class_instance = class_map[class_name]
                    instance = class_instance(**val)
                    all_objects = self.all()
                    all_objects[key] = instance
        except FileNotFoundError:
            pass

Analysis

• We open the file in read mode. This data would be loaded and transformed into something like our __objects dictionary (dictionary of objects).

• We loop through their key and value pairs which now looks like this

    {"BaseModel.b1806937-2f26-4053-aecd-ca259ea69ddd": 
            {"id": "b1806937-2f26-4053-aecd-ca259ea69ddd", 
             "created_at": "2024-02-09T12:06:45.830851", 
             "updated_at": "2024-02-09T12:06:45.830855", 
             "__class__": "BaseModel"},
     "User.a2f1fa1b-b5da-49f4-9338-a0b685482db0": 
            {"id": "a2f1fa1b-b5da-49f4-9338-a0b685482db0", 
             "created_at": "2024-02-09T12:07:49.418902", 
             "updated_at": "2024-02-09T12:07:49.418916", 
             "__class__": "User"}
     }
  

• We set the class name as it is.

• Then we get the instance of that class from our class_map dictionary and save in the class_instance variable. The class_map dictionary helps us to map various string to their respective models as we imported them. This is a very nice way to handle it such that we can recreate the instances of different classes as we import them.

• We recreate the instance. We are imagining something like so

    class_instance = class_map[class_name] # This would return let's say BaseModel
    instance = class_instance(**val) 
    # this is equivalent to
    instance = BaseModel(**val) # where val is the dictionary of key/value pairs of the particular key
  

• The instance variable is created with the **kwargs concept we learnt in the previous article.

• We call the self.all() which would give us a list of the already listed dictionaries and we save into the all_objects variable.

• Finally, we update the all_objects which would update the __objects dictionary by adding our new instance as an entry.

Reason for importing the modules in the function

The models were imported in the function, so as to prevent what is known as Circular Import.
Importing it in the function would ensure that, it is imported onlywhen that function is called.

Congratulations, now our storage functions have been successfully created. We would now tackle how to make our other class functions utilize this storage we have provided.

Updating the __init__.py file

Anytime we fun the program, we want our reload function to be kickstarted, so we get the saved data. Then our users can interact with this data, add or remove from them. In order to achieve this, we would have to add a block of code in the models/__init__.py file. The "init" file, would run automatically anytime the program is launched.

#!/usr/bin/python3
"""
Creating a unique File Storage instance for my application
"""
from models.engine.file_storage import FileStorage

storage = FileStorage()
storage.reload()

Analysis

• We import the FileStorage class from the "models/engine/file_storage" module.

• The FileStorage class is stored in an instance called storage.

• Using the new instance, we call the reload function to repopulate the data for our new program to be used.

Based on these changes, we would have to modify our BaseModel function as this is the function to which all the other classes would inherit.

Updating the BaseModel.py file

from models import storage # Add this to the imports

def __init__(self, *args, **kwargs):
        """
        Initialiazes new instance of BaseModel.

        Args:
            *args: Unused positional arguments
            **kwargs: Dictionary representation of an instance.

        If kwargs is not empty:
            Each key has an attribute name
            Each value is the value of the corresponding attr name
            Convert datetime to datetime objects

        Otherwise:
            Create id and created_at values as initially done
        """
        if kwargs:
            if '__class__' in kwargs:
                # Remove '__class__' from the dictionary
                del kwargs['__class__']
            if 'created_at' in kwargs:
                kwargs['created_at'] = datetime.strptime(
                        kwargs['created_at'], '%Y-%m-%dT%H:%M:%S.%f')
            if 'updated_at' in kwargs:
                kwargs['updated_at'] = datetime.strptime(
                        kwargs['updated_at'], '%Y-%m-%dT%H:%M:%S.%f')

            for key, value in kwargs.items():
                setattr(self, key, value)

        else:
            self.id = str(uuid.uuid4())
            self.created_at = datetime.now()
            self.updated_at = datetime.now()
            storage.new(self) # Added this line so that our initialization is saved in storage

def save(self):
        """
        Updates the public instance attribute update_at
        with the current datetime
        """
        self.updated_at = datetime.now()
        storage.save() """ Added this line so that the save function actually calls the save function
                        calls the save function in the storage to serialize this in JSON file"""

Conclusion

In the upcoming article, we'll take our project one step further by diving into the creation of the console interface using cmd.cmd. This interactive component will serve as a crucial tool for testing and interacting with our file storage system, bringing our project to life.

If you found this article helpful, show your support by leaving a like 💖, following the blog 👍🏾 and sharing your thoughts in the comments below. Your feedback fuels my journey!

Follow me on GitHub, let's get interactive on Twitter and form great connections on LinkedIn 😊

Happy coding 🥂