GLESGAE:The Resource Manager

Overview

Resources are everything that gets loaded, or generated, in your application. These can be anything from media objects - such as sound and textures - to more abstract things like level data, entities, or even the graphics system itself! So we need some way to describe what a Resource is, and a set of common functionality for dealing with them. Additionally, we probably want to manage these things, and loading/saving them would be a nicety too, if possible.

This section is all about that. We'll be building up a Resource Manager ( which has taken a fair amount of beating from myself to get right ) and a set of classes to describe Resources and Resource Banks - groups, if you will - that can be used to store and categorise our Resources.

Quick Start

We're upto revision 9 on the SVN now: svn co https://svn.xp-dev.com/svn/glesgae/trunk -r 9 ( though it's not currently live... - Stuckie 7th Feb 2012 )
For the brave, there's also the bleedin' edge git repository available: https://github.com/stuckie/glesgae though this may not always compile on anything but Linux.

What Is A Resource Manager?

Traditionally, there's usually just the one lone Resource Manager. This is generally tied in to the system's core file i/o and manages the loading and saving of pretty much everything. For systems such as Android where the entire GL Context can be lost at any time, having a Resource Manager which knows what has been loaded up - and more importantly in this case, how to reload it - is almost a requirement to stave off insanity.

Depending upon your view on Resources, the Manager can also tend to categorise Resources for easier access; for example keeping all Resources for rendering a Mesh close to each other so that the Renderer isn't hopping all over the memory to access things.

The GLESGAE Resource Manager

Our Resource Manager is going to be optional. It can be used, or it can be ignored. This is mostly because we haven't really written a set of file i/o utilities yet, so we're still either generating everything in code, or we're having Texture load up it's own bitmaps.. but also because there are times when a full blown Resource Manager is just too much for what we want to do. Having options is always nice.

Our Manager is going to manage Resource Banks - or groups of Resources, if you will. It'll be primarily responsible for the creation and management of banks of resources. This makes things much easier to deal with as when we're finished with file i/o and have all our formats defined, we can just tell the resource manager to load up huge blocks of data, and categorise them as needed.

Resource Banks

As stated, our Resource Banks will effectively act as fancy arrays of data. Each Bank is templated to a specific type of Resource, and can contain various groups, categorised into whatever we like. This means that we can have one Bank of Mesh objects, and group them specific to each Model, Level, whatever... so when we pull out a group to iterate over, they're all specific for that item. This will become important for when we get on to Entities and Components, as we'll be wanting to turn Components on and off, and the fastest way of processing anything to see if it's on or off, is to just not include the off objects in the same array as the on objects... effectively, a Resource Bank where the groups are Active and Inactive.

The Resource

The Resource itself is a relatively simple template class. It does follow a smart pointer-style setup, in that it tracks how many instances of itself there are, and only deletes the actual data it's holding when all instances are gone. It will also allow you to recast the Resource into other things - very handy because a Resource of a base class is a wholly different type than a Resource of a derived class from that base class.. so being able to recast to get back and forth is particularly handy - especially if the derived class has functionality the base does not.

Smart Pointers

Smart Pointers are named-so due to the fact they tend to track their instantiation, and only completely kill themselves when all instances are removed.

The Boost Library has shared_ptr, which is what our Resource is partly modelled on. You may be wondering why I didn't just keep Resource simple, and use smart pointers throughout anyway... this is because I do believe that while smart pointers are great for debugging, they're a bit large for normal use - and especially on more constrained hardware. So, I created Resource to imitate the bits I need.

Building our Resource

Our Resource Class needs to be a template. We have to be able to new our object directly into it, and then we can effectively forget about it. We also need to be able to track and find it again if we stash it somewhere. Of course, we can should also be able to ignore most of this functionality if need be.

The Hiccup

The current Resource code that's in the SVN is broken.. or at least, led to a more broken system. As such, we're going to be a bit confusing here and talk about the fixed system instead. The fixed system requires a few utility classes first, however... so we'll have a quick overview of them first.

The HashString

A HashString is effectively a number that's been generated from a string. As string methods can generally be rather expensive, we convert them to a number.. as equating numbers are much faster than strings, for instance. There's a caveat to this however, in that the conversion is a one-way process. For the most part, that's not a problem, it just makes debugging a bit more interesting when printing out a HashString gives out "60263687" instead of "Bob" ( for example )

There are many equations that can be used for HashStrings - from CRC, MD5, etc... though some have a bigger runtime hit than others. The one I settled upon is a slightly modified version of djb2. You can find it, some others, and more information here: http://www.cse.yorku.ca/~oz/hash.html

The HashString class can be found in the repository.

The Logger

I've also resurrected an old Logger class I did for a previous engine. It can spit out HTML formatted text, standard text, or to the console, and has INFO, DEBUG and ERROR log levels - to which DEBUG only shows if the DEBUG define is set. It could do with some sprucing up, but it's fine for our purposes for now.

The Logger class can also be found in the repository.

The Base Resource

As Resource itself is a templated class, we can't easily store a pointer to it. So, we do our usual trick of having a Base class that we can grab a pointer to, and redirect to the Template class anything that doesn't require the actual type.

We also store our Location information here, so we can find Resources again should we organise them into Groups and Banks.

#ifndef _BASE_RESOURCE_H_
#define _BASE_RESOURCE_H_

#include "../GLESGAETypes.h"
#include "../Utils/HashString.h"

namespace GLESGAE
{
	namespace Resources
	{
		typedef HashString Type;
		typedef unsigned int Id;
		typedef unsigned int Count;
		typedef unsigned int Group;
		
		struct Locator
		{
			Id bank;
			Type type;
			Group group;
			Id resource;
			
			Locator() : bank(INVALID), type(INVALID_HASHSTRING), group(INVALID), resource(INVALID) {}
		};
		
		// System Resources
		extern Type Camera;
		extern Type Controller;
		extern Type IndexBuffer;
		extern Type Material;
		extern Type Matrix2;
		extern Type Matrix3;
		extern Type Matrix4;
		extern Type Mesh;
		extern Type Shader;
		extern Type ShaderUniformUpdater;
		extern Type Texture;
		extern Type Timer;
		extern Type Vector2;
		extern Type Vector3;
		extern Type Vector4;
		extern Type VertexBuffer;
	}
	
	class BaseResourceBank;
	class BaseResource
	{
		public:			
			virtual ~BaseResource();
			
			/// Get the Type of this Resource
			Resources::Type getType() const { return mType; }
			
			/// Get which Group this Resource belongs to
			Resources::Group getGroup() const { return mGroup; }
			
			/// Get the Id of this Resource
			Resources::Id getId() const { return mId; }
			
			/// Get the Instance Count of this Resource
			Resources::Count getCount() const { return *mCount; }
						
		protected:
			/// Set the count
			void setCount(const Resources::Count& count) { *mCount = count; }
		
			/// Private constructor as this is a derived class only
			BaseResource(const Resources::Group group, const Resources::Type type, const Resources::Id id);
			
			/// Overloaded Copy Constructor, so we keep track of how many instances we have.
			BaseResource(const BaseResource& resource);
		
			/// Overloaded Assignment Operator, so we can keep track of everything.
			BaseResource& operator=(const BaseResource& resource)
			{
				mGroup = resource.mGroup;
				mType = resource.mType;
				mId = resource.mId;
				mCount = resource.mCount;
			
				return *this;
			}
	
			Resources::Group mGroup;
			Resources::Type mType;
			Resources::Id mId;
			mutable Resources::Count* mCount;
	};

}

#endif

So, what have we got in here....
We have a Locator struct. This'll be for sending into a Group or Bank so we can find a Resource again if need be. The more information we can fill in, the quicker the search will be.
We also define some types - Type, Id, Group and Count - which we define in the class itself; marking mCount as mutable so we can change it in const functions, the reason for which will become apparent in a minute!
Finally, we extern a bunch of Types for various system types which we have in the engine.. this is so we don't need to recalculate the HashString for it at run time - something which can turn out to be a costly procedure!

The Resource

With our BaseResource defined, we need our actual Resource class itself, which is where the magic happens.

#ifndef _RESOURCE_H_
#define _RESOURCE_H_

#include "BaseResource.h"
#include <cassert>

namespace GLESGAE
{
	template <typename T_Resource> class ResourceBank;
	template <typename T_Resource> class Resource : public BaseResource
	{
		// Resource Bank is a friend to access purge.
		friend class ResourceBank<T_Resource>;
		// It's a friend to itself to deal with the recast functionality.
		template <typename T_ResourceCast> friend class Resource;
		public:
			/// Dummy Constructor for creation of empty Resources.
			Resource() : BaseResource(INVALID, INVALID_HASHSTRING, INVALID), mResource(0) {}
			~Resource() { remove(); }
			
			/// Constructor for taking ownership over raw pointers.
			explicit Resource(T_Resource* const resource) : BaseResource(INVALID, INVALID_HASHSTRING, INVALID), mResource(resource) { instance(); }
			
			/// Pointer Operator overload to return the actual resource.
			T_Resource* operator-> () { return mResource; }
			
			/// Const Pointer Operator overload.
			T_Resource* operator-> () const { return mResource; }
			
			/// Dereference Operator overload to return the actual resource.
			T_Resource& operator* () { return *mResource; }
			
			/// Const Dereference operator overload.
			const T_Resource& operator* () const { return *mResource; }
			
			/// Recast Copy into a another Resource.
			template <typename T_ResourceCast> Resource<T_ResourceCast> recast()
			{
				Resource<T_ResourceCast> newResource(mGroup, mType, mId, reinterpret_cast<T_ResourceCast*>(mResource));
				delete newResource.mCount;
				newResource.mCount = mCount;
				instance();
				return newResource;
			}
			
			/// Recast Copy into a another Resource.
			template <typename T_ResourceCast> const Resource<T_ResourceCast> recast() const
			{
				Resource<T_ResourceCast> newResource(mGroup, mType, mId, reinterpret_cast<T_ResourceCast*>(mResource));
				delete newResource.mCount;
				newResource.mCount = mCount;
				instance();
				return newResource;
			}
			
			/// Overloaded Copy Constructor, so we keep track of how many instances we have.
			Resource(const Resource& resource)
			: BaseResource(resource)
			, mResource(resource.mResource)
			{
				instance();
			}
			
			/// Overloaded Assignment Operator to ensure we keep track of everything properly.
			Resource& operator=(const Resource& resource)
			{
				if (this != &resource) { // if someone's being daft and assigning ourselves, do nothing else we're likely to commit suicide.
					remove();
					
					BaseResource::operator=(resource);
					mResource = resource.mResource;
					
					instance();
				}
				
				return *this;
			}
			
			/// Overloaded Equals Operator for pointer checking.
			bool operator==(const Resource& resource) const
			{
				return (mResource == resource.mResource);
			}
			
			/// Overloaded Equals Operator for 0 pointer checking.
			bool operator==(const void* rhs) const
			{
				return (reinterpret_cast<void*>(mResource) == rhs);
			}
			
			/// Overloaded Not Equals Operator for pointer checking.
			bool operator!=(const Resource& resource) const
			{
				return !(*this == resource);
			}
			
			/// Overloaded Not Equals Operator for 0 pointer checking.
			bool operator!=(const void* rhs) const
			{
				return !(*this == rhs);
			}
			
			/// Increase the instance count of this Resource.
			/// Be exceptionally careful with using this manually, you will need to call remove manually as well!
			/// This is however, useful for anything that has to be sent a raw pointer which may leave C-scope.
			/// For example, Physics Engines and Scripting Languages.
			void instance() const
			{
				assert(mCount);
				++(*mCount);
			}

			/// Remove an instance count of this Resource, and if there are no more instances, purge it.
			/// Calling this manually should be used with caution, and only on a Resource which has been manually instanced.
			/// Otherwise, you will get into a situation whereby you've deleted something which still has a reference.
			/// Again, this is primarily useful for Physics Engines and Scripting Languages only.
			void remove()
			{
				assert(mCount);
				if ((*mCount) > 0U)
					--(*mCount);

				if ((*mCount) == 0U)
					purge();
			}
			
		protected:
			/// Protected Constructor so we can't create Managed Resources all over the place.
			explicit Resource(const Resources::Group group, const Resources::Type type, const Resources::Id id, T_Resource* const resource)
			: BaseResource(group, type, id)
			, mResource(resource)
			{
				instance();
			}
			
			/// Delete the actual resource.
			void purge() 
			{
				if (0 != mResource) {
					delete mResource; 
					mResource = 0;
				}
				
				if (0 != mCount) {
					delete mCount;
					mCount = 0;
				}
			}
			
		private:
			T_Resource* mResource;
	};	
}

#endif

Now, this is a beast of a class, so let's slowly walk through what we're doing here.

We have a bunch of constructors that do various things.
If we want to create an empty resource, for example, we can do something like Resource<Material> myMaterial; and myMaterial will effectively be a null pointer. This has many benefits as we can pre-allocate arrays of them, and not fill them in as yet, and use them as class variables that get filled later on and not necessarily in the constructor.
We can also feed a new Resource an already existing pointer: Material* myRawMaterial(new Material); Resource<Material> myMaterial(myRawMaterial); and Resource now manages myRawMaterial. The caveat here is that we should not delete myRawMaterial, as Resource will do it automatically for us when myMaterial goes out of scope.
We have overloaded the copy constructor and assignment operator so we can keep track of how many instances we have.
We also overload the pointer operator to give direct access to our data within, and do something slightly special with our equals operator, in that one of them takes a const void pointer. This is so that we can check whether our data is null or not.
Additionally, we can instance and remove ourselves if need be - sort of like the Obj-C retain and release ideology - but this can cause issues so should only be used if you know what you're doing!
Finally, we have a couple of special functions known as recast... which we use to recast a pointer to another. This is primarily for recasting up or down a class hierarchy - such as a Base Class to a Derived Class - as a Resource<BaseClass> is a completely different pointer to Resource<DerivedClass> even if DerivedClass is derived from BaseClass. This could be handy when DerivedClass offers additional functionality not found on BaseClass, but is platform specific.
And that's it really.. not that much of a scary class, it just does a lot of things.
You'll also see that the reason we made mCount mutable back in BaseResource, is that we need to modify it in our const recast function.

Building our Resource Banks

Resource Banks act as Managers for groups of Resources. Effectively, glorified arrays of specific types. They're again split into a Base class and a derived Template class.. so let's look at the Base class first.

The Base Resource Bank

#ifndef _BASE_RESOURCE_BANK_H_
#define _BASE_RESOURCE_BANK_H_

#include "BaseResource.h"

namespace GLESGAE
{
	class BaseResourceBank
	{
		friend class BaseResource;
		friend class ResourceManager;
		
		public:
			virtual ~BaseResourceBank() {} 
			
			/// Get the Type of this Resource Bank
			Resources::Type getType() const { return mType; }
			
			/// Get the If of this Resource Bank
			Resources::Id getId() const { return mId; }
			
			/// Create a new resource group.
			virtual Resources::Group newGroup() = 0;
			
			/// Remove Group
			virtual void removeGroup(const Resources::Group groupId) = 0;
			
			
		protected:
			/// Private constructor as this is a derived class only
			BaseResourceBank(const Resources::Id id, const Resources::Type type)
			: mId(id)
			, mType(type)
			{
			
			}
			
		private:
			// No Copying Allowed
			BaseResourceBank(const BaseResourceBank&);
			BaseResourceBank& operator=(const BaseResourceBank&);
			
			Resources::Id mId;
			Resources::Type mType;
	};

}

#endif

Quite a simple class, and bears much resemblance to BaseResource, in that it stores an Id and Type and not much else. One thing it does do, is provide interface ( pure virtual ) functions to create and remove groups which we will overload next.

The Resource Bank

#ifndef _RESOURCE_BANK_H_
#define _RESOURCE_BANK_H_

#include "Resource.h"
#include "BaseResourceBank.h"

#include <vector>
#include <cassert>

namespace GLESGAE
{
	template <typename T_Resource> class ResourceBank : public BaseResourceBank
	{
		// Resource is a friend to access instance.
		friend class Resource<T_Resource>;
		public:
			ResourceBank(const Resources::Id id, const Resources::Type type) : BaseResourceBank(id, type), mResources() {}
			~ResourceBank();
			
			/// Create a new resource group.
			Resources::Group newGroup();
			
			/// Get an entire Resource group.
			const std::vector<Resource<T_Resource> >& getGroup(const Resources::Group groupId) const;
			
			/// Remove Group
			void removeGroup(const Resources::Group groupId);
			
			/// Add a single Resource manually, and return the Resource version.
			Resource<T_Resource>& add(const Resources::Group groupId, const Resources::Type typeId, T_Resource* const resource);
			
			/// Add a group of Resources, and return the Group Id
			Resources::Group addGroup(const std::vector<Resource<T_Resource> >& resourceGroup);
			
			/// Get a Resource immediately
			Resource<T_Resource>& get(const Resources::Group groupId, const Resources::Id resourceId);	
			
		private:
			// Scary stuff... an array ( which we can access via Group Id) holding another array.
			// Second array holds the actual resource, where the id is it's array index.
			std::vector<std::vector<Resource<T_Resource> > > mResources;
	};
	
	template <typename T_Resource> ResourceBank<T_Resource>::~ResourceBank()
	{
		for (unsigned int index(0U); index < mResources.size(); ++index)
			removeGroup(index);
	}
		
	template <typename T_Resource> Resources::Group ResourceBank<T_Resource>::newGroup()
	{
		std::vector<Resource<T_Resource> > resourceArray;
		mResources.push_back(resourceArray);
	
		return mResources.size() - 1U;
	}
	
	template <typename T_Resource> const std::vector<Resource<T_Resource> >& ResourceBank<T_Resource>::getGroup(const Resources::Group groupId) const
	{
		// TODO: Scream if that groupId isn't valid, or doesn't exist...
		if (groupId == GLESGAE::INVALID) {
			assert(0);
			return;
		}
			
		return mResources[groupId];
	}

	template <typename T_Resource> void ResourceBank<T_Resource>::removeGroup(const Resources::Group groupId)
	{
		// TODO: Scream if that groupId isn't valid or doesn't exist...
		if (groupId == GLESGAE::INVALID)
			return;
	
		std::vector<Resource<T_Resource> >& resourceArray(mResources[groupId]);
		for (typename std::vector<Resource<T_Resource> >::iterator itr(resourceArray.begin()); itr < resourceArray.end(); ++itr) {
			if (itr->getCount() > 1U) {
				// TODO: scream bloody mary that there's still something using this resource.
			}
		}
	
		resourceArray.clear();
	}
	
	template <typename T_Resource> Resource<T_Resource>& ResourceBank<T_Resource>::add(const Resources::Group groupId, const Resources::Type typeId, T_Resource* const resource)
	{
		std::vector<Resource<T_Resource> >& resourceArray(mResources[groupId]);
		const Resources::Id resourceId(resourceArray.size());
	
		resourceArray.push_back(Resource<T_Resource>(groupId, typeId, resourceId, resource));
		
		return resourceArray[resourceId];
	}
	
	template <typename T_Resource> Resources::Group ResourceBank<T_Resource>::addGroup(const std::vector<Resource<T_Resource> >& resourceGroup)
	{
		const Resources::Group groupId(mResources.size());
		
		mResources.push_back(resourceGroup);
		return groupId;
	}

	template <typename T_Resource> Resource<T_Resource>& ResourceBank<T_Resource>::get(const Resources::Group groupId, const Resources::Id resourceId)
	{
		assert(groupId != GLESGAE::INVALID);
		assert(resourceId != GLESGAE::INVALID);
		return mResources[groupId][resourceId];
	}

}


#endif

The actual Resource Bank itself is a bit more complicated, as it does all the work.
You'll also notice usage of assert everywhere, this is good defensive programming and should be used often! This is why the Resource system ended up in such a mess in the first place - I wasn't using enough asserts to catch things, and memory was being overwritten all over the place.

Anyway, the basic crux of the Resource Manager, is that it stores an array of arrays.
These are indexed firstly by the groupId to find the correct group, followed by the resourceId to find the correct Resource. If you remember our Locator struct we defined in BaseResource, that's how we can find things when needed.

One interesting thing is we also take in the Type and an Id for the Resource Bank. This is because we can have many Resource Banks attached to the Resource Manager, so each needs it's own Id. The Type is used for double checking, even though we template it to a specific class/struct type, we need to know it's HashString type to be able to search for them without requiring to know it's template type.

Building our Resource Manager

Our last piece of the system, is the Resource Manager itself.
This isn't much different from the Resource Bank above, in that it's more a wrapper around managing arrays.

The Resource Manager

#ifndef _RESOURCE_MANAGER_H_
#define _RESOURCE_MANAGER_H_

#include "ResourceBank.h"

#include <string>
#include <map>
#include <cassert>

namespace GLESGAE
{
	class ResourceManager
	{
		friend class BaseResource;
		public:
			ResourceManager() : mResourceBanks() {}
			~ResourceManager() { assert(mResourceBanks.empty()); }

			/// Create a new Resource Bank of the specified
			template <typename T_Resource> ResourceBank<T_Resource>& createBank(const Resources::Type bankType);
			
			/// Retrieve a Resource Bank
			template <typename T_Resource> ResourceBank<T_Resource>& getBank(const Resources::Id bankId, const Resources::Type bankType);
			
			/// Delete a Resource Bank
			template <typename T_Resource> void removeBank(const Resources::Id bankId, const Resources::Type bankType);
			
			/// Load Resource Bank from Disk
			template <typename T_Resource> void loadBankResources(const std::string& bankSet, const Resources::Id bankId, const Resources::Type bankType);
			
			/// Save Resource Bank to Disk
			template <typename T_Resource> void saveBankResources(const std::string& bankSet, const Resources::Id bankId, const Resources::Type bankType);
			
		private:
			// A map denoting the name of a resource bank, and the resource bank pointer itself.
			std::map<Resources::Id, BaseResourceBank*> mResourceBanks;
	};
	
	template <typename T_Resource> ResourceBank<T_Resource>& ResourceManager::createBank(const Resources::Type bankType)
	{
		// TODO: check bank doesn't already exist.
		Resources::Id bankId = mResourceBanks.size();
		mResourceBanks[bankId] = new ResourceBank<T_Resource>(bankId, bankType);
		return *(reinterpret_cast<ResourceBank<T_Resource>*>(mResourceBanks[bankId]));
	}
	
	template <typename T_Resource> ResourceBank<T_Resource>& ResourceManager::getBank(const Resources::Id bankId, const Resources::Type)
	{
		// TODO: check bank actually exists.
		assert(bankId != INVALID);
		return *(reinterpret_cast<ResourceBank<T_Resource>*>(mResourceBanks[bankId]));
	}
	
	template <typename T_Resource> void ResourceManager::removeBank(const Resources::Id bankId, const Resources::Type /*typeId*/)
	{
		std::map<Resources::Id, BaseResourceBank*>::iterator bank(mResourceBanks.find(bankId));
		if (bank != mResourceBanks.end()) {
			// TODO: Check that the bankType matches up!
			delete reinterpret_cast<ResourceBank<T_Resource>*>(bank->second);
			bank->second = 0;
		}
	
		// TODO: Error that we can't find this bank!
	}
/*	
	template <typename T_Resource> void ResourceManager::loadBankResources(const std::string& bankSet, const Resources::Id bankId, const Resources::Type bankType)
	{
	}
	
	template <typename T_Resource> void ResourceManager::saveBankResources(const std::string& bankSet, const Resources::Id bankId, const Resources::Type bankType)
	{
	}
*/
}

#endif

There are a few oddities with this class.
Firstly, it's a bit incomplete ( the load/save functionality ) as we still have to write proper File utilities.
Secondly, the destructor doesn't actually wipe out any Resource Banks that may still be left around. The reason for this is that you really should be doing this manually. We do assert if it's not empty, however... and further functionality will have us going over this class to add in the load/save as well as outputting which banks have been left behind for the user to fix and clean up.

Conclusion

A bit of a long one this time.. but we needed to get the entire Resource System out in one go.
As we can see.. we can use Resource<T> on it's own, or use it in conjunction with the just Resource Banks, or the entire Resource System.
We have a helper struct to be able to pass around where Resources can be found if we need to find them elsewhere. This may seem a bit useless for now, but as it's a POD-style struct, we can use these for when we do start to load and save the Resource Banks.

Next up, we'll be looking into the State System, and then onto some Scripting.