Sorted by Squirrels.

Sunday, 13 April 2008

u-Fat filesystem

Not a truncated attempt at a verbal slight, but a solution to PC compatible file access for micro-controllers!

Many FAT file system implementations are, by necessity, too large to take a place in micro-controller applications. Long file names, fragmented files, the FAT tables - no wonder a 'proper' solution weighs in at many K of code and buffers. One solution is to carry out raw access to the card, but this then involves compromises when getting the data off the card onto the desktop.

My solution is to reduce weight of code by applying stringent use restrictions:

Only 16 files (potentially), named in 8.3 format, pre-allocated and contiguous.

This enables us to locate the start sector of the file and read/write to it as we wish. The files can be freely swapped between PC and micro controller application using standard operating system commands. To reach this state of nirvana all that needs to be done is format the card before copying the pre-allocated files to it.

The only external requirement for servicing the code is a function that reads sectors, using logical block addressing, from your chosen FAT device. If you look in the ACE post previously the code zip there includes a complete MMC solution.

I found a schematic on the arduino site that is almost identical to my own and comes very highly recommended. If you haven't got a card reader that you can cannibalise then edge connector works well too. It's a common way of doing it and there are many good examples around - you can find one here.

Here's the header which shows just how micro micro can be!

 /*
/ Microfat.
/
/ Developed by Charlie Robson in 2008.
/
/ This is the bare minimum* functionality required to read and write data
/ to a FAT formatted device. In this case, an MMC card.
/
/ Rather than build a fully-featured FAT implimentation this will allow
/ the user to locate sectors associated with a file on the device, and
/ read and write to them in a brute force manner. No error checking, no
/ long filenames, not even any support for fragmented files :) This is
/ raw access, we only deal with existing files.
/
/ What you do get is a very lightweight module for extremely limited
/ environments. Arduino has 2k of SRAM. Holding the sector of FAT data
/ needed to traverse a fragged file would instantly gobble up 1/4
/ of that. Enough said. Another benefit to not using the FAT is that
/ there's no need to worry about whether a device is FAT12 or 16.
/
/ To use:
/ Format a card.
/ Make a file big enough to hold all the data you expect to write.
/ Copy to the card.
/
/ With the MMC module started up, call initialise. This will cache some
/ relevant information about the card such as the sector location of the
/ root directory and data area.
/
/ Call locateFileStart, passing the name of the file which is providing
/ the backing store for your transfers, and assuming the file is found
/ in the directory you'll receive its start sector and file size.
/
/ Now feel free to read and write data to and from the sectors allocated
/ to the file. Code carefully though - stray writes can twat your card!
/
/ Enjoy :)
/
/ *Massive understatement!
*/

#ifndef __MICROFAT_H__
#define __MICROFAT_H__

// some data structures we'll be encountering on our travels.

typedef struct
{
byte bootable;
byte chsAddrOfFirstSector[3];
byte partitionType;
byte chsAddrOfLastSector[3];
uint32_t lbaAddrOfFirstSector;
uint32_t partitionLengthSectors;
}
partition_record;

typedef struct
{
byte jump[3];
char oemName[8];
uint16_t bytesPerSector;
byte sectorsPerCluster;
uint16_t reservedSectors;
byte fatCopies;
uint16_t rootDirectoryEntries;
uint16_t totalFilesystemSectors;
byte mediaDescriptor;
uint16_t sectorsPerFAT;
uint16_t sectorsPerTrack;
uint16_t headCount;

uint32_t hiddenSectors;
uint32_t totalFilesystemSectors2;
byte logicalDriveNum;
byte reserved;
byte extendedSignature;
uint32_t partitionSerialNum;
char volumeLabel[11];
char fsType[8];
byte bootstrapCode[447];
byte signature[2];
}
boot_sector;

typedef struct
{
char filespec[11];
byte attributes;
byte reserved[10];
uint16_t time;
uint16_t date;
uint16_t startCluster;
uint32_t fileSize;
}
directory_entry;


namespace microfat
{
// Cache some relevant info about the card.
//
bool initialise(byte* buffer);

// Get start sector, file size for given filename. Returns false if the file
// is not found in the directory.
//
bool locateFileStart(const char* filename, unsigned long& sector, unsigned long& size);
};


#endif // __MICROFAT_H__



Now for the meat:

#include <wprogram.h>
#include "microfat.h"
#include "mmc.h"

// Data which remains constant over one session. Re-initialise if
// the card is changed.
//
static struct
{
uint16_t sectorsPerCluster;

uint32_t rootDirSect, cluster2;

byte* buffer;
}
vars;


bool microfat::initialise(byte* buffer)
{
vars.buffer = buffer;

if (RES_OK != mmc::readSectors(vars.buffer, 0, 1))
{
return false;
}

partition_record* p = (partition_record*)&vars.buffer[0x1be];
long bootSector = p->lbaAddrOfFirstSector;

if (RES_OK != mmc::readSectors(vars.buffer, bootSector, 1))
{
return false;
}

boot_sector* b = (boot_sector*)vars.buffer;

if (BYTESPERSECTOR != b->bytesPerSector)
{
return false;
}

vars.sectorsPerCluster = b->sectorsPerCluster;
vars.rootDirSect = bootSector + b->reservedSectors + (b->fatCopies * b->sectorsPerFAT);

long dirBytes = b->rootDirectoryEntries * 32;
long dirSects = dirBytes / BYTESPERSECTOR;

if (dirBytes % BYTESPERSECTOR != 0)
{
++dirSects;
}

vars.cluster2 = vars.rootDirSect + dirSects;

return true;
}



// Get start sector for given filename.
//
// Returns false if the specified file wasn't found in the directory.
// Short filenames only. Be aware that the directory fills up with 'deleted'
// filenames. Rather than deleting too many files try a quick reformat.
//
bool microfat::locateFileStart(const char* filename, uint32_t& sector, uint32_t& size)
{
if (RES_OK == mmc::readSectors(vars.buffer, vars.rootDirSect, 1))
{
// The filenames are stored in [8][3] format. No dot, all upper case.
// Names shorter than 8 chars are padded with spaces.
//
// Cook the supplied name. fred.txt -> FRED----TXT
//
char cookedName[11];
for(int i = 0; i < 12; ++i)
{
cookedName[i] = 0x20;
}

for (int i = 0, j = 0; i < 12 && filename[i]; ++i)
{
if (filename[i] != '.')
{
// Force char to uppercase. Cheesy I know :)
//
cookedName[j] = filename[i] >= 96 ? filename[i] - 32 : filename[i];
++j;
}
else
{
// Continue cooking chars at the extension position
//
j = 8;
}
}

for (int i = 0; i < BYTESPERSECTOR; i += 32)
{
directory_entry* de = (directory_entry*)&vars.buffer[i];

// Don't match with deleted or system/volname/subdir/hidden files
//
if (de->filespec[0] != 0xe5 && (de->attributes & 0x1e) == 0 && memcmp(cookedName, de->filespec, 11) == 0)
{
sector = vars.cluster2 + ((de->startCluster-2) * vars.sectorsPerCluster);
size = de->fileSize;
return true;
}
}
}

return false;
}




Which as you can see is smaller than the header - due in no small measure to the size of the FAT structure definitions.

As it stands this code could be improved by using non-static file info structures - only one is supported right now but the change would be trivial. Also allowing the filename search to traverse the entire root directory space would allow access to more files. However I don't really see the utility in ths -wherever I've used this code I've never needed more than one file :)

As usual any Qs are welcome.

12 comments:

Dinesh said...

Hi there!

I am interested in your uFat library for use on an Arduino. I'm pretty new to digital i/o, so I have some simple questions:

1. Where does one get the mmc.h library? Is there a common one floating around out there somewhere?

2. Once I have the start sector and file size information from locateFileStart, do I just do raw sector writes and reads? Is there a good library around for this? I've seen some people using modified version of the Roland Reigel library.

Thanks for the help!

Sir Morris said...

Hi Dinesh. Everything you need is here.

http://groups.google.com/group/micro-hacker

Get the file mmc1.rar. It's a small demo that locates a file fred.txt on a card, and writes into the 1st 512 bytes of it.

Email me if you have any queries and I'll try to help!

peter said...

in microfat::locateFileStart

you have got

char cookedName[11];
for(int i = 0; i < 12; ++i)
{
cookedName[i] = 0x20;
}

The size of the cookedName array is filled with spaces beyond the end.

Change 11 to 12, or 12 to 11 might be OK to

/Peter Lorenzen

Sir Morris said...

Well spotted! A schoolboy error on my part :)

Look out for uFat 2.0..

AprendizDeTodo said...

In order to compile in arduino 12 add #include "WProgram.h" en mmc.h

Cheers

Mat said...

Is it possible to use UFAT library to actually create a file on the SD card? Like copying a file from the pc and pasting it directly into the SD card (connected to Arduino)?

Thanks

Sir Morris said...

Hi Mat,

Sorry - no. The file must already exist and hold enough data for you to overwrite.

Anonymous said...

Hi There,

This looks like the perfect solution to my card reading problem but I cant find the mcc1.rar file on your google group page...

Where can I find this file?

Thanks

Sir Morris said...

Hello Anonymous! How nice to hear from you again.

There were further developments rendering the old code obsolete. Continue reading the blog or skip to the ufat2 post.

Enjoy!

Anonymous said...

>> edge connector works well too

another cheap/n/easy way to connect is to use a micro-SD and solder wires to an SD-sized SD-to-micro-SD adaptor (these often come free with SD cards).

Nice article/code, thanks,

Neal.

Alex Yalo said...

Hi, really nice work back in 2008.
I am working on an arduino project and need a library like yours but with the ability to read files with long names and more than 16. I'm going crazy trying to find it but cannot, maybe do you know any libary that lets me do that?
Thanks a lot

Sir Morris said...

Hi,

for proper file system support you need to investigate something like chan's fat library. This does everything but comes at the price of being more complex.

http://elm-chan.org/fsw/ff/00index_e.html

HTH

This is how we do it

MMC (9) acorn atom (7) zx81 (7) arduino (5) Atari 800 (3) c128 (3) sd2iec (3) tatung einstein (3) 6502 (2) Chuckie egg (2) Max6956 (2) QL (2) RCM (2) avr (2) c64 (2) cadsoft eagle (2) eeprom (2) mmbeeb (2) sd card (2) sio2sd (2) spi (2) ufat2 (2) vic20 (2) 6502 second processor (1) 6522 (1) 8255 (1) Acorn BBC Micro (1) Apple 2e (1) Apple ][ 2 two (1) BBC 6502 second processor (1) BBC micro (1) DevicePrint (1) Double Choc Chip Muffins (1) FAT (1) IO (1) Jupiter Ace (1) LED (1) Master 128 (1) PCB (1) PIC (1) POV (1) PROGMEM (1) ST (1) Spectrum 128 (1) antex (1) arcade spinner (1) arduino shield (1) assembler (1) atmel (1) bakewell tart (1) beer (1) bird's nest (1) cake (1) cassette (1) cassette interface (1) compact flash (1) dac (1) de-yellowing (1) efficient (1) einSDein (1) eye strain (1) failosophy (1) filesystem (1) fram (1) french polishing (1) fuse (1) fuses (1) gaming (1) glue (1) google chrome (1) heroic failure (1) high voltage programming (1) hot irons (1) if (1) jiffydos (1) joey beltram (1) lego robot (1) library (1) lying (1) matron (1) microcode (1) mmc interface (1) mmc2iec (1) mmm (1) mouse guts (1) oscilloscopes (1) pcm (1) pic32mx (1) porn (1) proto shield (1) retro computer museum (1) retro hard-on (1) rom box (1) sd (1) sd2mmc (1) seadragon (1) silliness (1) small (1) software master (1) soldering (1) spi software master (1) stray capacitance (1) string (1) techadventure (1) test equipment porn (1) ts1000 (1) turtle cheesecake (1) tweaking (1) vc20 (1) video head (1) video ram replacement (1) weewee (1) wingasm (1) wire library (1) wodges of IO (1) xilinx cpld (1) yellowing (1) zx spectrum (1) zxpander (1)

About Sir Morris

My Photo

Loves: Old computers, Old Techno, Old ladies. Cake.
Hates: New computers.

Unless otherwise stated all of the original work presented here is:

Creative Commons License
Licensed under a Creative Commons Attribution-Noncommercial 2.5 Generic License.

The work of others where referenced will be attributed appropriately. If I've failed to do this please let me know.