FiST: Stackable File System Language and Templates

File System development is very difficult and time consuming. Even small changes to existing file systems require deep understanding of kernel internals, making the barrier to entry for new developers high. Moreover, porting file system code from one operating system to another is almost as difficult as the first port. Past proposals to provide extensible (stackable) file system interfaces would have simplified the development of new file systems. These proposals, however, advocated massive changes to existing operating system interfaces and existing file systems; operating system vendors and maintainers resist making any large changes to their kernels because of stability and performance concerns. As a result, file system development is still a difficult, long, and non-portable process.

The FiST (File System Translator) system combines two methods to solve the above problems in a novel way: a set of stackable file system templates for each operating system, and a high-level language that can describe stackable file systems in a cross-platform portable fashion. Using FiST, stackable file systems need only be described once. FiST's code generation tool, fistgen, compiles a single file system description into loadable kernel modules for several operating systems (currently Solaris, Linux, and FreeBSD). The project demonstrates that with FiST, code size and development time are reduced significantly, while imposing a small performance overhead of only 1-2%. These benefits are achieved, as well as portability, without changing existing operating systems or file system.

Public Software:

Mailing Lists:

Click on the list name below to access the subscription Web page. That page will also tell you how to post to the list, access the lists' archives, change your subscription status, etc.

List Name Purpose Who Can Subscribe Who Can Post Comments
fistgen For users' and developers' questions, announcements, bug reports, and general. Anyone Subscribers only This list used to be called fist at cs.columbia.edu. Use new name please.
fistgen-announce For announcements only Anyone Developers only  

Miscellaneous:

If you are interested in a stackable file systems in general, I have a collection of old stacking papers.

If you find a problem, please describe it in detail and submit a bug report via Bugzilla.

Download software.

Journal Articles:

# Title (click for html version) Formats Published In Date Comments
1 The Case for Specialized File Systems, or, Fighting File System Obesity PDF BibTeX ;login: The USENIX Magazine Feb 2010  
2 On Incremental File System Development PS PDF BibTeX ACM Transactions on Storage (TOS) May 2006  
3 Versatility and Unix Semantics in Namespace Unification PS PDF BibTeX ACM Transactions on Storage (TOS) Feb 2006  
4 Unionfs: Bringing File Systems Together BibTeX Linux Journal Dec 2004  
5 Writing Stackable Filesystems BibTeX Linux Journal May 2003 Software described in article

Conference and Workshop Papers:

# Title (click for html version) Formats Published In Date Comments
1 RAIF: Redundant Array of Independent Filesystems PS PDF BibTeX IEEE Conference on Mass Storage Systems and Technologies (MSST 2007) Sep 2007  
2 Kernel Support for Stackable File Systems PS PDF BibTeX 2007 Ottawa Linux Symposium Jun 2007  
3 Making Linux Stacking-Friendly BibTeX First Linux Storage and Filesystem Workshop, held in conjuction with the fifth USENIX Conference on File and Storage Technologies (FAST 2007) Feb 2007  
4 UnionFS: User- and Community-oriented Development of a Unification Filesystem PS PDF BibTeX 2006 Ottawa Linux Symposium Jul 2006  
5 Accurate and Efficient Replaying of File System Traces PS PDF BibTeX Fourth USENIX Conference on File and Storage Technologies (FAST 2005) Dec 2005  
6 Adding Secure Deletion to Your Favorite File System PS PDF BibTeX Third IEEE Security In Storage Workshop (SISW 2005) Dec 2005  
7 Increasing Distributed Storage Survivability with a Stackable RAID-like File System PS PDF BibTeX First IEEE/ACM Workshop on Cluster Security, in conjunction with the Fifth IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2005) May 2005 Won Best Paper Award
8 I3FS: An In-Kernel Integrity Checker and Intrusion Detection File System PS PDF BibTeX Usenix Large Installation Systems Administration Conference (LISA 18) Nov 2004  
9 Avfs: An On-Access Anti-Virus File System PS PDF BibTeX 13th USENIX Security Symposium (Security 2004) Aug 2004  
10 A Versatile and User-Oriented Versioning File System PS PDF BibTeX Third USENIX Conference on File and Storage Technologies (FAST 2004) Apr 2004  
11 Tracefs: A File System to Trace Them All PS PDF BibTeX Third USENIX Conference on File and Storage Technologies (FAST 2004) Apr 2004  
12 Reducing Storage Management Costs via Informed User-Based Policies PS PDF BibTeX NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST 2004) Apr 2004  
13 Cryptographic File Systems Performance: What You Don't Know Can Hurt You PS PDF BibTeX 2003 IEEE Security In Storage Workshop (SISW 2003) Oct 2003  
14 NCryptfs: A Secure and Convenient Cryptographic File System PS PDF BibTeX Slides Usenix Technical Conference, General Track Jun 2003  
15 Fast Indexing: Support for Size-Changing Algorithms in Stackable File System PS PDF BibTeX Usenix Technical Conference Jun 2001  
16 FiST: A Language for Stackable File Systems PS PDF BibTeX Usenix Technical Conference Jun 2000  
17 Extending File Systems Using Stackable Templates PS PDF BibTeX Usenix Technical Conference Jun 1999  
18 A Stackable File System Interface For Linux PS PDF BibTeX LinuxExpo 99 May 1999  

Technical Reports:

# Title (click for html version) Formats Published In Date Comments
1 Storage Virtualization with a Stackable File System BibTeX Stony Brook U. CS TechReport FSL-05-03 Dec 2005 M.S. Thesis
2 Versatility and Unix Semantics in a Fan-Out Unification File System PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-01b Oct 2004  
3 Versatile File System Tracing with Tracefs PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-05 Aug 2004 M.S. Thesis
4 Enhancing File System Integrity Through Checksums PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-04 May 2004  
5 Operating System Support for Extensible Secure File Systems PS PDF BibTeX Stony Brook U. CS TechReport FSL-04-02 May 2004 Ph.D. Research Proficiency Exam (RPE)
6 Cryptographic File Systems Performance: What You Don't Know Can Hurt You PS PDF BibTeX Stony Brook U. CS TechReport FSL-03-02 Aug 2003 Slightly expanded version of SISW'03 paper with same title.
7 Reducing Storage Management Costs via Informed User-Based Policies PS PDF BibTeX Stony Brook U. CS TechReport FSL-03-01 Mar 2003  
8 The Design and Implementation of Elastic Quotas: A System for Flexible File System Management PS PDF BibTeX Columbia U. CS TechReport CUCS-014-02 Jun 2002  
9 FiST: A System for Stackable File System Code Generation PS PDF BibTeX Columbia University May 2001 Ph.D. Dissertation
10 Stackable File Systems As a Security Tool PS PDF BibTeX Columbia U. CS TechReport CUCS-036-99 Dec 1999  
11 Usenetfs: A Stackable File System for Large Article Directories PS PDF BibTeX Columbia U. CS TechReport CUCS-022-98 Jul 1998  
12 Cryptfs: A Stackable Vnode Level Encryption File System PS PDF BibTeX Columbia U. CS TechReport CUCS-021-98 Jun 1998 This tech-report had been cited more than 15 times before the new version of the work was published -- NCryptfs.
13 FiST: A File System Component Compiler PS PDF BibTeX Columbia University Apr 1997 Ph.D. Thesis Proposal

Works In Progress:

# Title (click for html version) Formats Published In Date Comments
1 Fast Indexing: Support for Size-Changing Algorithms in Stackable File System PS PDF BibTeX   Oct 2001 Expanded version of Usenix 2001 paper

Past Students:

# Name (click for home page) Program Period Current Location
1 Charles P. Wright PhD May 2003 - May 2006 Application Software Developer, Walleye Software (New York, NY)
2 Shrikar Archak MS Sep 2009 - Dec 2010 Senior Software Engineer, Core Deduplication Filesystem group, Datadomain/EMC
3 Puja Gupta MS Jan 2003 - Dec 2003 File Systems Engineer, Apple (Cupertino, CA)
4 Harikesavan Pathangi Krishnan MS Jan 2003 - Dec 2003 Software Engineer, Engineering department for WAFS product line, Packeteer, Inc. (South Plainfield, NJ)
5 Kiran-Kumar Muniswamy-Reddy MS Jan 2002 - May 2004 Harvard University CS Ph.D. program (Cambridge, MA)
6 Sunil Satnur MS Sep 2004 - Dec 2005 Senior Member of the Technical Staff, Performance and Storage groups, VMware, Inc. (Palo Alto, CA)
7 Kiron Vijayasankar MS Dec 2006 - Dec 2007 Member of Technical Staff, Engineering Development, Riverbed Technology (Sunnyvale, CA)
8 Charles P. Wright BS (Honors) Dec 2001 - May 2003 Software Developer, Eladian Partners, LLC (New York, NY)
9 Josef "Jeff" Sipek BS Jul 2005 - May 2008 Nexenta Systems (Ann Arbor, MI)

Sponsors:

# Sponsor Amount Period Type Title (click for award abstract)
1 HP/Intel $131,529 2002-2003 Sole PI Linux Application Performance and File System Security
2 HP/Intel $22,490 2001-2002 Sole PI Linux Network Scalability and File System Reliability


(Last updated: Mon Feb 17 17:12:08 EST 2014)