Fuel

Fuel is a general purpose open-source framework to serialize and deserialize objects, based on a pickling algorithm which is implemented in Pharo. We demonstrate that it is possible to build a really fast and full-featured serializer without specific VM support, with a clean object-oriented design.

You can follow our news in the blog.

Presentations and publications

In these slides we show the pickle format and the serialization algorithm main idea with an example.

We presented Fuel at ESUG International Workshop on Smalltalk Technologies 2011. You can watch the presentation either in YouTube or mp4. You can also get the slides either in pdf or slideshare.

In proceedings of this workshop we published a short paper about Fuel which includes detailed description and benchmarks: Clustered Serialization with Fuel. (Martin Dias, Mariano Martinez Peck, Stéphane Ducasse and Gabriela Arévalo)

An extended version of that paper is under preparation, which will add, among others, more detailed benchmark analysis like presence/absence of JIT, memory consumption, etc.

We also presented the project at Smalltalks 2011. You can see the slides.

Blog posts, screencasts and videos

Apart from our own blog, you can read interesting articles about Fuel in Mariano's blog.

We truly appreciate James Robertson's screencast that shows how to download and try Fuel. In addition, James explains in another screencast how to use Fuel Package Loader to export and import all seaside packages. Here Pavel Krivanek shows how he can export PharoCore with Fuel and materialize it in 15 seconds in a PharoKernel image.

Goals

Fast: We worry a lot about to have the best performance. We developed a complete benchmark suite to help analyse the performance with diverse sample sets, as well as compare against other serializers. The pickling algorithm of Fuel allows outstanding materialization performance as well as very good serialization performance too.

Well designed: From the start was a constraint to have a good object-oriented design and to do not need any special support from the VM. Fuel has a lot of test cases, with a high coverage. We also worry about writing comments on classes and methods.

Concrete: We don't have a dialect-interchange format aspiration. Despite we have ports to other dialects, the development is Pharo-centric. This enables us to serialize special objects like contexts, block closures, exceptions, compiled methods and classes.

Flexible: Depending on the context, there could be multiple ways of serialize the same object. For example, a class can be considered either a global or a regular object. In the former case, it will be encoded just its name; in the latter case, the class will be encoded in detail, with its method dictionary, etc.

Features

Support class reshape (when the class of serialized objects has changed).
Serialize any kind of object in the image.
Complete serialization of classes and traits (not just as global names).
Support cycles and avoid duplicates in the graph.
Global objects references: when you serialize Transcript or the system dictionary, it is not duplicated and instead managed as a global reference.
Support graph substitutions ("for serializing this object, instead take this one").
Solve common problems like Set and Dictionary rehash.
Buffered writing: we use a buffered write stream for the serialization part (thanks Sven!).

Future

Continue efforts on performance optimization.
Replace ImageSegment, SmartRefStream, ReferenceStream and DataStream.
Improve the tool for loading packages.
Fast statistics/brief info extraction of a stored graph.
Partial loading of a stored graph.
Many more in our issue tracker.

Software using Fuel

Many software currently uses Fuel, we are happy with the community feedback. For detailed information please see Software using Fuel.

Documentation

Developer guides

Install

Evaluate in a Pharo environment:

Gofer it
        squeaksource: 'MetacelloRepository';
        package: 'ConfigurationOfFuel';
        load.
((Smalltalk at: #ConfigurationOfFuel) project version: '1.7') load.

Try it

| sourceArray loadedArray |
sourceArray := 
	Array 
		with: 'a string' 
		with: Transcript
		with: [ Transcript show: 'a string' ].

"Store to the file"
FLSerializer serialize: sourceArray toFileNamed: 'example.FL'.

"Load from the file"
loadedArray := FLMaterializer materializeFromFileNamed: 'example.FL'.

"The arrays are not the same"
[ sourceArray ~~ loadedArray ] assert.

"The strings are not the same"
[ sourceArray first ~~ loadedArray first ] assert.
[ sourceArray first = loadedArray first ] assert.

"The global instance Transcript is the same"
[ sourceArray second == loadedArray second ] assert.

"Look at Transcript how the loaded block prints a string"
loadedArray third value.

Versions

Current version: 1.7

Announcement

We have released the new version 1.7 of Fuel. The list of changes includes performance optimizations, design clean-ups, and new features.

The FLSerializer and FLMaterializer API has changed. They are no longer implementing the algorithms but are a kind of Façade.
Developer guides with examples: Getting Started and Customizing the graph.
Serialization substitutions: "Store this object instead of me."
Global sends: "Restore me by sending this selector to this global"
Versioning the stream: We prefix the stream with a version number that should match when loading.
Performance optimizations on instances of:
- Word-like classes. (We thank a lot to Henrik Sperre Johansen for your help!)
- ByteString and Symbol.
- Date, Time, Duration and DateAndTime.
- Point and Rectangle.
- MethodDictionary. Now materialization is 2000x faster, thanks to its new rehash without become.
Huge clean-up in Tests package.