Introduction
************

This document describes the implementation of the fmc bus for Linux.
FMC (FPGA Mezzanine Card) is the standard we use for our I/O devices, in
the context of White Rabbit and related hardware.

In our I/O environments we need to write drivers for each mezzanine
card, and such drivers must work regardless of the carrier being used.
To achieve this, we abstract the FMC interface.

We have a carrier for PCI-E called SPEC and one for VME called SVEC, but
more are planned.  Also, we support stand-alone devices (usually plugged
on a SPEC card), controlled through Etherbone, developed by GSI.

Code and documentation for the FMC bus was born as part of the spec-sw
project, but now it lives in its own project.  Other projects, i.e.
software support for the various carriers, should include this as a
submodule.

The most up to date version of code and documentation is always
available from the repository you can clone from:

        git://ohwr.org/fmc-projects/fmc-bus.git (read-only)
        git@ohwr.org:fmc-projects/fmc-bus.git (read-write for developers)

Selected versions of the documentation, as well as complete tar archives
for selected revisions are placed to the Files section of the project:
<http://www.ohwr.org/projects/fmc-bus/files>

1 Basic Concepts
****************

2 What is FMC
*************

FMC, as said, stands for "FPGA Mezzanine Card".  It is a standard
developed by the VME consortium called VITA (VMEbus International Trade
Association and ratified by ANSI, the American National Standard
Institute.  The official documentation is called "ANSI-VITA 57.1".

The FMC card is an almost square PCB, around 70x75 millimeters, that is
called mezzanine in this document.  It usually lives plugged into
another PCB for power supply and control; such bigger circuit board is
called carrier from now on, and a single carrier may host more than one
mezzanine.

In the typical application the mezzanine is mostly analog while the
carrier is mostly digital, and hosts an FPGA that must be configured to
match the specific mezzanine and the desired application.  Thus, you may
need to load different FPGA images to drive different instances of the
same mezzanine.

FMC, as such, is not a bus in the usual meaning of the term, because
most carriers have only one connector, and carriers with several
connectors have completely separate electrical connections to them.
This package, however, implements a bus as a software abstraction.

3 What is a Linux Bus
*********************

Within the Linux kernel, a bus is a data structure with a few methods.
It's main role is registering a list of devices and a list of drivers,
offering a match function that compares the respective identifiers (in a
bus-specific way) to assign drivers to devices.

Activation and deactivation of devices happens through the probe and
remove functions of the respective driver; an advanced user can also use
sysfs to change the binding of drivers to devices (for example, if more
than one driver can drive the same device you may want to force the
choice).

4 What is SDB
*************

SDB (Self Describing Bus) is a set of data structures that we use for
enumerating the internal structure of an FPGA image.  We also use it as
a filesystem inside the FMC EEPROM.

SDB is not mandatory for use of this FMC kernel bus, but if you have SDB
this package can make good use of it.  SDB itself is developed in the
fpga-config-space OHWR project.  The link to the repository is
<git://ohwr.org/hdl-core-lib/fpga-config-space.git> and what is used in
this project lives in the sdbfs subdirectory in there.

SDB support for FMC is described in *note FMC Identification:: and *note
SDB Support::

5 Functions Exported by fmc.ko
******************************

The FMC core exports the usual 4 functions that are needed for a bus to
work, and a few more:

        int fmc_driver_register(struct fmc_driver *drv);
        void fmc_driver_unregister(struct fmc_driver *drv);
        int fmc_device_register(struct fmc_device *fmc);
        void fmc_device_unregister(struct fmc_device *fmc);

        int fmc_device_register_n(struct fmc_device **fmc, int n);
        void fmc_device_unregister_n(struct fmc_device **fmc, int n);

        uint32_t fmc_readl(struct fmc_device *fmc, int offset);
        void fmc_writel(struct fmc_device *fmc, uint32_t val, int off);
        void *fmc_get_drvdata(struct fmc_device *fmc);
        void fmc_set_drvdata(struct fmc_device *fmc, void *data);

        int fmc_reprogram(struct fmc_device *f, struct fmc_driver *d, char *gw,
                          int sdb_entry);

The data structure that describe a device is detailed in *note FMC
Device::, the one that describes a driver is detailed in *note FMC
Driver::.  Please note that structures of type fmc_device must be
allocated by the caller, but must not be released after unregistering.
The fmc-bus itself takes care of releasing the structure when their use
count reaches zero - actually, the device model does that in lieu of us.

The functions to register and unregister n devices are meant to be used
by carriers that host more than one mezzanine.  The devices must all be
registered at the same time because if the FPGA is reprogrammed, all
devices in the array are affected.  Usually, the driver matching the
first device will reprogram the FPGA, so other devices must know they
are already driven by a reprogrammed FPGA.

If a carrier hosts slots that are driven by different FPGA devices, it
should register as a group only mezzanines that are driven by the same
FPGA, for the reason outlined above.

Finally, the fmc_reprogram function calls the reprogram method (see
*note The API Offered by Carriers:: and also scans the memory area for
an SDB tree.  You can pass -1 as sdb_entry to disable such scan.
Otherwise, the function fails if no tree is found at the specified entry
point.  The function is meant to factorize common code, and by the time
you read this it is already used by the spec-sw and fine-delay modules.

6 Module Parameters in fmc.ko
*****************************

The core driver receives two module parameters, meant to help debugging
client modules.  Both parameters can be modified by writing to
/sys/module/fmc/parameters/, because they are used when client drivers
are devices are registered, not when fmc.ko is loaded.

'dump_eeprom='

     If not zero, the parameter asks the bus controller to dump the
     EEPROM of any device that is registered, using printk.

'dump_sdb='

     If not zero, the parameter prints the SDB tree of every FPGA it is
     loaded by fmc_reprogram().  If greater than one, it asks to dump
     the binary content of SDB records.  This currently only dumps the
     top-level SDB array, though.

EEPROM dumping avoids repeating lines, since most of the contents is
usually empty and all bits are one or zero.  This is an example of the
output:

        [ 6625.850480] spec 0000:02:00.0: FPGA programming successful
        [ 6626.139949] spec 0000:02:00.0: Manufacturer: CERN
        [ 6626.144666] spec 0000:02:00.0: Product name: FmcDelay1ns4cha
        [ 6626.150370] FMC: mezzanine 0: 0000:02:00.0 on SPEC
        [ 6626.155179] FMC: dumping eeprom 0x2000 (8192) bytes
        [ 6626.160087] 0000: 01 00 00 01  00 0b 00 f3  01 0a 00 a5  85 87 c4 43
        [ 6626.167069] 0010: 45 52 4e cf  46 6d 63 44  65 6c 61 79  31 6e 73 34
        [ 6626.174019] 0020: 63 68 61 c7  70 72 6f 74  6f 2d 30 cc  45 44 41 2d
        [ 6626.180975] 0030: 30 32 32 36  37 2d 56 33  da 32 30 31  32 2d 31 31
        [...]
        [ 6626.371366] 0200: 66 64 65 6c  61 79 0a 00  00 00 00 00  00 00 00 00
        [ 6626.378359] 0210: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00
        [ 6626.385361] [...]
        [ 6626.387308] 1800: 70 6c 61 63  65 68 6f 6c  64 65 72 ff  ff ff ff ff
        [ 6626.394259] 1810: ff ff ff ff  ff ff ff ff  ff ff ff ff  ff ff ff ff
        [ 6626.401250] [...]

The dump of SDB looks like the following; the example shows the simple
golden gateware for the SPEC card, removing the leading timestamps to
fit the page:

        spec 0000:02:00.0: SDB: 00000651:e6a542c9 WB4-Crossbar-GSI
        spec 0000:02:00.0: SDB: 0000ce42:ff07fc47 WR-Periph-Syscon (00000000-000000ff)
        FMC: mezzanine 0: 0000:02:00.0 on SPEC
        FMC: poor dump of sdb first level:
        0000: 53 44 42 2d  00 02 01 00  00 00 00 00  00 00 00 00
        0010: 00 00 00 00  00 00 01 ff  00 00 00 00  00 00 06 51
        0020: e6 a5 42 c9  00 00 00 02  20 12 05 11  57 42 34 2d
        0030: 43 72 6f 73  73 62 61 72  2d 47 53 49  20 20 20 00
        0040: 00 00 01 01  00 00 00 07  00 00 00 00  00 00 00 00
        0050: 00 00 00 00  00 00 00 ff  00 00 00 00  00 00 ce 42
        0060: ff 07 fc 47  00 00 00 01  20 12 03 05  57 52 2d 50
        0070: 65 72 69 70  68 2d 53 79  73 63 6f 6e  20 20 20 01

7 FMC Device
************

Within the Linux bus framework, the FMC device is created and registered
by the carrier driver.  For example, the PCI driver for the SPEC card
fills a data structure for each SPEC that it drives, and registers an
associated FMC device for each card.  The SVEC driver can do exactly the
same for the VME carrier (actually, it should do it twice, because the
SVEC carries two FMC mezzanines).  Similarly, an Etherbone driver will
be able to register its own FMC devices, offering communication
primitives through frame exchange.

The contents of the EEPROM within the FMC are used for identification
purposes, i.e.  for matching the device with its own driver.  For this
reason the device structure includes a complete copy of the EEPROM
(actually, the carrier driver may choose whether or not to return it -
for example we most likely won't have the whole EEPROM available for
Etherbone devices.

The following listing shows the current structure defining a device.
Please note that all the machinery is in place but some details may
still change in the future.  For this reason, there is a version field
at the beginning of the structure.  As usual, the minor number will
change for compatible changes (like a new flag) and the major number
will increase when an incompatible change happens (for example, a change
in layout of some fmc data structures).  Device writers should just set
it to the value FMC_VERSION, and be ready to get back -EINVAL at
registration time.

     struct fmc_device {
             unsigned long version;
             unsigned long flags;
             struct module *owner;           /* char device must pin it */
             struct fmc_fru_id id;           /* for EEPROM-based match */
             struct fmc_operations *op;      /* carrier-provided */
             int irq;                        /* according to host bus. 0 == none */
             int eeprom_len;                 /* Usually 8kB, may be less */
             int eeprom_addr;                /* 0x50, 0x52 etc */
             uint8_t *eeprom;                /* Full contents or leading part */
             char *carrier_name;             /* "SPEC" or similar, for special use */
             void *carrier_data;             /* "struct spec *" or equivalent */
             __iomem void *fpga_base;        /* May be NULL (Etherbone) */
             __iomem void *slot_base;        /* Set by the driver */
             struct fmc_device **devarray;   /* Allocated by the bus */
             int slot_id;                    /* Index in the slot array */
             int nr_slots;                   /* Number of slots in this carrier */
             unsigned long memlen;           /* Used for the char device */
             struct device dev;              /* For Linux use */
             struct device *hwdev;           /* The underlying hardware device */
             unsigned long sdbfs_entry;
             struct sdb_array *sdb;
             uint32_t device_id;             /* Filled by the device */
             char *mezzanine_name;           /* Defaults to ``fmc'' */
             void *mezzanine_data;
     };

The meaning of most fields is summarized in the code comment above.

The following fields must be filled by the carrier driver before
registration:

   * version: must be set to FMC_VERSION.
   * owner: set to MODULE_OWNER.
   * op: the operations to act on the device.
   * irq: number for the mezzanine; may be zero.
   * eeprom_len: length of the following array.
   * eeprom_addr: 0x50 for first mezzanine and so on.
   * eeprom: the full content of the I2C EEPROM.
   * carrier_name.
   * carrier_data: a unique pointer for the carrier.
   * fpga_base: the I/O memory address (may be NULL).
   * slot_id: the index of this slot (starting from zero).
   * memlen: if fpga_base is valid, the length of I/O memory.
   * hwdev: to be used in some dev_err() calls.
   * device_id: a slot-specific unique integer number.

Please note that the carrier should read its own EEPROM memory before
registering the device, as well as fill all other fields listed above.

The following fields should not be assigned, because they are filled
later by either the bus or the device driver:

   * flags.
   * fru_id: filled by the bus, parsing the eeprom.
   * slot_base: filled and used by the driver, if useful to it.
   * devarray: an array og all mezzanines driven by a singe FPGA.
   * nr_slots: set by the core at registration time.
   * dev: used by Linux.
   * sdb: FPGA contents, scanned according to driver's directions.
   * sdbfs_entry: SDB entry point in EEPROM: autodetected.
   * mezzanine_data: available for the driver.
   * mezzanine_name: filled by fmc-bus during identification.

Note: mezzanine_data may be redundant, because Linux offers the drvdata
approach, so the field may be removed in later versions of this bus
implementation.

As I write this, she SPEC carrier is already completely functional in
the fmc-bus environment, and is a good reference to look at.

7.1 The API Offered by Carriers
===============================

The carrier provides a number of methods by means of the
'fmc_operations' structure, which currently is defined like this (again,
it is a moving target, please refer to the header rather than this
document):

     struct fmc_operations {
             uint32_t (*readl)(struct fmc_device *fmc, int offset);
             void (*writel)(struct fmc_device *fmc, uint32_t value, int offset);
             int (*reprogram)(struct fmc_device *f, struct fmc_driver *d, char *gw);
             int (*validate)(struct fmc_device *fmc, struct fmc_driver *drv);
             int (*irq_request)(struct fmc_device *fmc, irq_handler_t h,
                                char *name, int flags);
             void (*irq_ack)(struct fmc_device *fmc);
             int (*irq_free)(struct fmc_device *fmc);
             int (*gpio_config)(struct fmc_device *fmc, struct fmc_gpio *gpio,
                                int ngpio);
             int (*read_ee)(struct fmc_device *fmc, int pos, void *d, int l);
             int (*write_ee)(struct fmc_device *fmc, int pos, const void *d, int l);
     };

The individual methods perform the following tasks:

'readl'
'writel'

     These functions access FPGA registers by whatever means the carrier
     offers.  They are not expected to fail, and most of the time they
     will just make a memory access to the host bus.  If the carrier
     provides a fpga_base pointer, the driver may use direct access
     through that pointer.  For this reason the header offers the inline
     functions fmc_readl and fmc_writel that access fpga_base if the
     respective method is NULL. A driver that wants to be portable and
     efficient should use fmc_readl and fmc_writel.  For Etherbone, or
     other non-local carriers, error-management is still to be defined.

'validate'

     Module parameters are used to manage different applications for two
     or more boards of the same kind.  Validation is based on the busid
     module parameter, if provided, and returns the matching index in
     the associated array.  See *note Module Parameters:: in in doubt.
     If no match is found, '-ENOENT' is returned; if the user didn't
     pass 'busid=', all devices will pass validation.  The value
     returned by the validate method can be used as index into other
     parameters (for example, some drivers use the 'lm32=' parameter in
     this way).  Such "generic parameters" are documented in *note
     Module Parameters::, below.  The validate method is used by
     'fmc-trivial.ko', described in *note fmc-trivial::.

'reprogram'

     The carrier enumerates FMC devices by loading a standard (or
     golden) FPGA binary that allows EEPROM access.  Each driver, then,
     will need to reprogram the FPGA by calling this function.  If the
     name argument is NULL, the carrier should reprogram the golden
     binary.  If the gateware name has been overridden through module
     parameters (in a carrier-specific way) the file loaded will match
     the parameters.  Per-device gateware names can be specified using
     the 'gateware=' parameter, see *note Module Parameters::.  Note:
     Clients should call rhe new helper, fmc_reprogram, which both calls
     this method and parse the SDB tree of the FPGA.

'irq_request'
'irq_ack'
'irq_free'

     Interrupt management is carrier-specific, so it is abstracted as
     operations.  The interrupt number is listed in the device
     structure, and for the mezzanine driver the number is only
     informative.  The handler will receive the fmc pointer as dev_id;
     the flags argument is passed to the Linux request_irq function, but
     fmc-specific flags may be added in the future.  You'll most likely
     want to pass the 'IRQF_SHARED' flag.

'gpio_config'

     The method allows to configure a GPIO pin in the carrier, and read
     its current value if it is configured as input.  See *note The GPIO
     Abstraction:: for details.

'read_ee'
'write_ee'

     Read or write the EEPROM.  The functions are expected to be only
     called before reprogramming and the carrier should refuse them with
     'ENODEV' after reprogramming.  The offset is expected to be within
     8kB (the current size), but addresses up to 1MB are reserved to fit
     bigger I2C devices in the future.  Carriers may offer access to
     other internal flash memories using these same methods: for example
     the SPEC driver may define that its carrier I2C memory is seen at
     offset 1M and the internal SPI flash is seen at offset 16M. This
     multiplexing of several flash memories in the same address space is
     is carrier-specific and should only be used by a driver that has
     verified the 'carrier_name' field.

7.2 The GPIO Abstraction
========================

Support for GPIO pins in the fmc-bus environment is not very
straightforward and deserves special discussion.

While the general idea of a carrier-independent driver seems to fly,
configuration of specific signals within the carrier needs at least some
knowledge of the carrier itself.  For this reason, the specific driver
can request to configure carrier-specific GPIO pins, numbered from 0 to
at most 4095.  Configuration is performed by passing a pointer to an
array of struct fmc_gpio items, as well as the length of the array.
This is the data structure:

        struct fmc_gpio {
                char *carrier_name;
                int gpio;
                int _gpio;      /* internal use by the carrier */
                int mode;       /* GPIOF_DIR_OUT etc, from <linux/gpio.h> */
                int irqmode;    /* IRQF_TRIGGER_LOW and so on */
        };

By specifying a carrier_name for each pin, the driver may access
different pins in different carriers.  The gpio_config method is
expected to return the number of pins successfully configured, ignoring
requests for other carriers.  However, if no pin is configured (because
no structure at all refers to the current carrier_name), the operation
returns an error so the caller will know that it is running under a
yet-unsupported carrier.

So, for example, a driver that has been developed and tested on both the
SPEC and the SVEC may request configuration of two different GPIO pins,
and expect one such configuration to succeed - if none succeeds it most
likely means that the current carrier is a still-unknown one.

If, however, your GPIO pin has a specific known role, you can pass a
special number in the gpio field, using one of the following macros:

        #define FMC_GPIO_RAW(x)         (x)             /* 4096 of them */
        #define FMC_GPIO_IRQ(x)         ((x) + 0x1000)  /*  256 of them */
        #define FMC_GPIO_LED(x)         ((x) + 0x1100)  /*  256 of them */
        #define FMC_GPIO_KEY(x)         ((x) + 0x1200)  /*  256 of them */
        #define FMC_GPIO_TP(x)          ((x) + 0x1300)  /*  256 of them */
        #define FMC_GPIO_USER(x)        ((x) + 0x1400)  /*  256 of them */

Use of virtual GPIO numbers (anything but FMC_GPIO_RAW) is allowed
provided the carrier_name field in the data structure is left
unspecified (NULL). Each carrier is responsible for providing a mapping
between virtual and physical GPIO numbers.  The carrier may then use the
_gpio field to cache the result of this mapping.

All carriers must map their I/O lines to the sets above starting from
zero.  The SPEC, for example, maps interrupt pins 0 and 1, and test
points 0 through 3 (even if the test points on the PCB are called
5,6,7,8).

If, for example, a driver requires a free LED and a test point (for a
scope probe to be plugged at some point during development) it may ask
for FMC_GPIO_LED(0) and FMC_GPIO_TP(0).  Each carrier will provide
suitable GPIO pins.  Clearly, the person running the drivers will know
the order used by the specific carrier driver in assigning leds and
testpoints, so to make a carrier-dependent use of the diagnostic tools.

In theory, some form of autodetection should be possible: a driver like
the wr-nic (which uses IRQ(1) on the SPEC card) should configure IRQ(0),
make a test with software-generated interrupts and configure IRQ(1) if
the test fails.  This probing step should be used because even if the
wr-nic gateware is known to use IRQ1 on the SPEC, the driver should be
carrier-independent and thus use IRQ(0) as a first bet - actually, the
knowledge that IRQ0 may fail is carrier-dependent information, but using
it doesn't make the driver unsuitable for other carriers.

The return value of gpio_config is defined as follows:

   * If no pin in the array can be used by the carrier, '-ENODEV'.
   * If at least one virtual GPIO number cannot be mapped, '-ENOENT'.
   * On success, 0 or positive.  The value returned is the number of
     high input bits (if no input is configured, the value for success
     is 0).

While I admit the procedure is not completely straightforward, it allows
configuration, input and output with a single carrier operation.  Given
the typical use case of FMC devices, GPIO operations are not expected to
ever by in hot paths, and GPIO access so fare has only been used to
configure the interrupt pin, mode and polarity.  Especially reading
inputs is not expected to be common.  If your device has GPIO
capabilities in the hot path, you should consider using the kernel's
GPIO mechanisms.

7.3 fmc-fakedev
===============

This package includes a software-only device, called fmc-fakedev, which
is able to register up to 4 mezzanines (by default it registers one).
Unlike the SPEC driver, which creates an FMC device for each PCI cards
it manages, this module creates a single instance of its set of
mezzanines.

It is meant as the simplest possible example of how a driver should be
written, and it includes a fake EEPROM image (built using the tools
described in *note FMC Identification::), which by default is replicated
for each fake mezzanine.

You can also use this device to verify the match algorithms, by asking
it to test your own EEPROM image.  You can provide the image by means of
the eeprom= module parameter: the new EEPROM image is loaded, as usual,
by means of the firmware loader.  This example shows the defaults and a
custom EEPROM image:

        spusa.root# insmod fmc-fakedev.ko
        [   99.971247]  fake-fmc-carrier: mezzanine 0
        [   99.975393]       Manufacturer: fake-vendor
        [   99.979624]       Product name: fake-design-for-testing
        spusa.root# rmmod fmc-fakedev
        spusa.root# insmod fmc-fakedev.ko eeprom=fdelay-eeprom.bin
        [  121.447464]  fake-fmc-carrier: Mezzanine 0: eeprom "fdelay-eeprom.bin"
        [  121.462725]  fake-fmc-carrier: mezzanine 0
        [  121.466858]       Manufacturer: CERN
        [  121.470477]       Product name: FmcDelay1ns4cha
        spusa.root# rmmod fmc-fakedev

After loading the device, you can use the write_ee method do modify its
own internal fake EEPROM: whenever the image is overwritten starting at
offset 0, the module will unregister and register again the FMC device.
This is shown later, in section *note fmc-write-eeprom::.

In order to register more than one mezzanine, you can use module
parameters:

'ndev=<number>'

     Ask for a number of devices.  By default it registers one device or
     as many as EEPROM file names it receives.

'eeprom=<name>[,<name>]'

     Name of the EEPROM images to be faked in the mezzanine devices.
     You can pass up to 4 comma-separated names; empty names or trailing
     unspecified names will select the default built-in EEPROM image.
     If you specify more than one EEPROM name, the module will
     automatically register more than one mezzanine.

7.4 FMC Device Incompatibilities
================================

During development, we had to introduce a few incompatibilities in the
data structures.  By using the field named version inside fmc_device you
should be notified about such changes.  Please use git blame for the
details - the version is always updated in the same commit that makes
the change (for example commit 14a7b580 introduced version 3, changed
the data structures and documentation at the same time).  See *note FMC
Device:: for the current list of fields.

The other incompatible change we had to introduce is in the allocation
policy for fmc_device structures.  The devices must be allocated one at
a time (with kmalloc) and never released.  Releasing happens by means of
the release method of the device, when its use count drops to zero.

As a side effect, fmc_device_register_n must be passed an array of
pointers, not an array of devices.  Since each device is freed
separately when not in use any more.

8 FMC Driver
************

An FMC driver is concerned with the specific mezzanine and associated
gateware.  As such, it is expected to be independent of the carrier
being used: it will perform I/O accesses only by means of
carrier-provided functions.

The matching between device and driver is based on the content of the
EEPROM (as mandated by the FMC standard) or by the actual cores
configured in the FPGA; the latter technique is used when the FPGA is
already programmed when the device is registered to the bus core.

In some special cases it is possible for a driver to directly access
FPGA registers, by means of the 'fpga_base' field of the device
structure.  This may be needed for high-bandwidth peripherals like fast
ADC cards.  If the device module registered a remote device (for example
by means of Etherbone), the 'fpga_base' pointer will be NULL. Therefore,
drivers must be ready to deal with NULL base pointers, and fail
gracefully.  Most driver, however, are not expected to access the
pointer directly but run fmc_readl and fmc_writel instead, which will
work in any case.

In even more special cases, the driver may access carrier-specific
functionality: the 'carrier_name' string allows the driver to check
which is the current carrier and make use of the 'carrier_data' pointer.
We chose to use carrier names rather than numeric identifiers for
greater flexibility, but also to avoid a central registry within the
'fmc.h' file - we hope other users will exploit our framework with their
own carriers.  An example use of carrier names is in GPIO setup (see
*note The GPIO Abstraction::), although the name match is not expected
to be performed by the driver.  If you depend on specific carriers,
please check the carrier name and fail gracefully if your driver finds
it is running in a yet-unknown-to-it environment.

8.1 ID Table
============

Like most other Linux drivers, and FMC driver must list all the devices
which it is able to drive.  This is usually done by means of a device
table, but in FMC we can match hardware based either on the contents of
their EEPROM or on the actual FPGA cores that can be enumerated.
Therefore, we have two tables of identifiers.

Matching of FRU information depends on two names, the manufacturer (or
vendor) and the device (see *note FMC Identification::); for flexibility
during production (i.e.  before writing to the EEPROM) the bus supports
a catch-all driver that specifies NULL strings.  For this reason, the
table is specified as pointer-and-length, not a a null-terminated array
- the entry with NULL names can be a valid entry.

Matching on FPGA cores depends on two numeric fields: the 64-bit vendor
number and the 32-bit device number.  Support for matching based on
class is not yet implemented.  Each device is expected to be uniquely
identified by an array of cores (it matches if all of the cores are
instantiated), and for consistency the list is passed as
pointer-and-length.  Several similar devices can be driven by the same
driver, and thus the driver specifies and array of such arrays.

The complete set of involved data structures is thus the following:

        struct fmc_fru_id { char *manufacturer; char *product_name; };
        struct fmc_sdb_one_id { uint64_t vendor; uint32_t device; };
        struct fmc_sdb_id { struct fmc_sdb_one_id *cores; int cores_nr; };

        struct fmc_device_id {
                struct fmc_fru_id *fru_id; int fru_id_nr;
                struct fmc_sdb_id *sdb_id; int sdb_id_nr;
        };

A better reference, with full explanation, is the <linux/fmc.h> header.

8.2 Module Parameters
=====================

Most of the FMC drivers need the same set of kernel parameters.  This
package includes support to implement common parameters by means of
fields in the 'fmc_driver' structure and simple macro definitions.

The parameters are carrier-specific, in that they rely on the busid
concept, that varies among carriers.  For the SPEC, the identifier is a
PCI bus and devfn number, 16 bits wide in total; drivers for other
carriers will most likely offer something similar but not identical, and
some code duplication is unavoidable.

This is the list of parameters that are common to several modules to see
how they are actually used, please look at spec-trivial.c.

'busid='

     This is an array of integers, listing carrier-specific
     identification numbers.  For PIC, for example, '0x0400' represents
     bus 4, slot 0.  If any such ID is specified, the driver will only
     accept to drive cards that appear in the list (even if the FMC ID
     matches).  This is accomplished by the validate carrier method.

'gateware='

     The argument is an array of strings.  If no busid= is specified,
     the first string of gateware= is used for all cards; otherwise the
     identifiers and gateware names are paired one by one, in the order
     specified.

'show_sdb='

     For modules supporting it, this parameter asks to show the SDB
     internal structure by means of kernel messages.  It is disabled by
     default because those lines tend to hide more important messages,
     if you look at the system console while loading the drivers.  Note:
     the parameter is being obsoleted, because fmc.ko itself now
     supports dump_sdb= that applies to every client driver.

For example, if you are using the trivial driver to load two different
gateware files to two different cards, you can use the following
parameters to load different binaries to the cards, after looking up the
PCI identifiers.  This has been tested with a SPEC carrier.

        insmod fmc-trivial.ko \
                              busid=0x0200,0x0400 \
                              gateware=fmc/fine-delay.bin,fmc/simple-dio.bin

Please note that not all sub-modules support all of those parameters.
You can use modinfo to check what is supported by each module.

8.3 fmc-trivial
===============

The simple module fmc-trivial is just a simple client that registers an
interrupt handler.  I used it to verify the basic mechanism of the FMC
bus and how interrupts worked.

The module implements the generic FMC parameters, so it can program a
different gateware file in each card.  The whole list of parameters it
accepts are:

'busid='
'gateware='
     Generic parameters.  See *note Module Parameters::.

This driver is worth reading, but it is not worth describing here.

8.4 fmc-write-eeprom
====================

This module is designed to load a binary file from /lib/firmware and to
write it to the internal EEPROM of the mezzanine card.  This driver uses
the 'busid' generic parameter.

Overwriting the EEPROM is not something you should do daily, and it is
expected to only happen during manufacturing.  For this reason, the
module makes it unlikely for the random user to change a working EEPROM.

However, since the EEPROM may include application-specific information
other than the identification, later versions of this packages added
write-support through sysfs.  See *note Accessing the EEPROM::.

To avoid damaging the EEPROM content, the module takes the following
measures:

   * It accepts a 'file=' argument (within /lib/firmware) and if no such
     argument is received, it doesn't write anything to EEPROM (i.e.
     there is no default file name).

   * If the file name ends with '.bin' it is written verbatim starting
     at offset 0.

   * If the file name ends with '.tlv' it is interpreted as
     type-length-value (i.e., it allows writev(2)-like operation).

   * If the file name doesn't match any of the patterns above, it is
     ignored and no write is performed.

   * Only cards listed with 'busid=' are written to.  If no busid is
     specified, no programming is done (and the probe function of the
     driver will fail).

Each TLV tuple is formatted in this way: the header is 5 bytes, followed
by data.  The first byte is 'w' for write, the next two bytes represent
the address, in little-endian byte order, and the next two represent the
data length, in little-endian order.  The length does not include the
header (it is the actual number of bytes to be written).

This is a real example: that writes 5 bytes at position 0x110:

        spusa.root# od -t x1 -Ax /lib/firmware/try.tlv
        000000 77 10 01 05 00 30 31 32 33 34
        00000a
        spusa.root# insmod /tmp/fmc-write-eeprom.ko busid=0x0200 file=try.tlv
        [19983.391498] spec 0000:03:00.0: write 5 bytes at 0x0110
        [19983.414615] spec 0000:03:00.0: write_eeprom: success

Please note that you'll most likely want to use SDBFS to build your
EEPROM image, at least if your mezzanines are being used in the White
Rabbit environment.  For this reason the TLV format is not expected to
be used much and is not expected to be developed further.

If you want to try reflashing fake EEPROM devices, you can use the
fmc-fakedev.ko module (see *note fmc-fakedev::).  Whenever you change
the image starting at offset 0, it will deregister and register again
after two seconds.  Please note, however, that if fmc-write-eeprom is
still loaded, the system will associate it to the new device, which will
be reprogrammed and thus will be unloaded after two seconds.  The
following example removes the module after it reflashed fakedev the
first time.

     spusa.root# insmod fmc-fakedev.ko
        [   72.984733]  fake-fmc: Manufacturer: fake-vendor
        [   72.989434]  fake-fmc: Product name: fake-design-for-testing
        spusa.root# insmod fmc-write-eeprom.ko busid=0 file=fdelay-eeprom.bin; \
            rmmod fmc-write-eeprom
        [  130.874098]  fake-fmc: Matching a generic driver (no ID)
        [  130.887845]  fake-fmc: programming 6155 bytes
        [  130.894567]  fake-fmc: write_eeprom: success
        [  132.895794]  fake-fmc: Manufacturer: CERN
        [  132.899872]  fake-fmc: Product name: FmcDelay1ns4cha

8.5 fmc-chardev
===============

This is a simple generic driver, that allows user access by means of a
character device (actually, one for each mezzanine it takes hold of).

The char device is created as a misc device.  Its name in /dev (as
created by udev) is the same name as the underlying FMC device.  Thus,
the name can be a silly fmc-0000 look-alike if the device has no
identifiers nor bus_id, a more specific fmc-0400 if the device has a
bus-specific address but no associated name, or something like
fdelay-0400 if the FMC core can rely on both a mezzanine name and a bus
address.

Currently the driver only supports read and write: you can lseek to the
desired address and read or write a register.

The driver assumes all registers are 32-bit in size, and only accepts a
single read or write per system call.  However, as a result of Unix read
and write semantics, users can simply fread or fwrite bigger areas in
order to dump or store bigger memory areas.

There is currently no support for mmap, user-space interrupt management
and DMA buffers.  They may be added in later versions, if the need
arises.

The example below shows raw access to a SPEC card programmed with its
golden FPGA file, that features an SDB structure at offset 256 - i.e.
64 words.  The mezzanine's EEPROM in this case is not programmed, so the
default name is fmc-<bus><devfn>, and there are two cards in the system:

         spusa.root# insmod fmc-chardev.ko
         [ 1073.339332] spec 0000:02:00.0: Driver has no ID: matches all
         [ 1073.345051] spec 0000:02:00.0: Created misc device "fmc-0200"
         [ 1073.350821] spec 0000:04:00.0: Driver has no ID: matches all
         [ 1073.356525] spec 0000:04:00.0: Created misc device "fmc-0400"
         spusa.root# ls -l /dev/fmc*
         crw------- 1 root root 10, 58 Nov 20 19:23 /dev/fmc-0200
         crw------- 1 root root 10, 57 Nov 20 19:23 /dev/fmc-0400
         spusa.root# dd bs=4 skip=64 count=1 if=/dev/fmc-0200 2> /dev/null | od -t x1z
         0000000 2d 42 44 53                                      >-BDS<
         0000004

The simple program tools/fmc-mem in this package can access an FMC char
device and read or write a word or a whole area.  Actually, the program
is not specific to FMC at all, it just uses lseek, read and write.

Its first argument is the device name, the second the offset, the third
(if any) the value to write and the optional last argument that must
begin with "+" is the number of bytes to read or write.  In case of
repeated reading data is written to stdout; repeated writes read from
stdin and the value argument is ignored.

The following examples show reading the SDB magic number and the first
SDB record from a SPEC device programmed with its golden image:

        spusa.root# ./fmc-mem /dev/fmc-0200 100
        5344422d
        spusa.root# ./fmc-mem /dev/fmc-0200 100 +40 | od -Ax -t x1z
        000000 2d 42 44 53 00 01 02 00 00 00 00 00 00 00 00 00  >-BDS............<
        000010 00 00 00 00 ff 01 00 00 00 00 00 00 51 06 00 00  >............Q...<
        000020 c9 42 a5 e6 02 00 00 00 11 05 12 20 2d 34 42 57  >.B......... -4BW<
        000030 73 6f 72 43 72 61 62 73 49 53 47 2d 00 20 20 20  >sorCrabsISG-.   <
        000040

9 Writing your FMC Kernel Module
********************************

This chapter includes some suggestions about to write your own drivers
for FMC carriers or mezzanines.  It is the outcome of the experience I
gathered working with fine-delay and 'wr-nic.ko' (currently part of
spec-sw, as I write this).

Driver writing is a known problem, which only requires you to know your
hardware and your frameworks.  However, the choice to split the overall
FMC software design into several projects, and thus several
repositories, opens a new set of problems, related to how the projects
relate one another.

A typical carrier driver will be built against a specific version of
this fmc-bus package.  A mezzanine driver, in addition, will most often
be developed under a specific carrier and will most likely depend on
features of the the specific carrier, at least initially.

To make the problem worse, a single host in your operating environment
("in production") may need tp run several FMC devices (thus several FMC
drivers), that expect to be using different versions of the base
packages - according to when and where they have been written.

9.1 Versions of the Bus Abstraction
===================================

This fmc-bus package includes a version field at the beginning of the
two data structures, 'fmc_device' and 'fmc_driver'.

This version number is there to prevent mismatches between the
information passed by the FMC modules and how it is used by the core
code.  The version is not expected to change often, because the code
base is already pretty stable, and we don't expect to face many new
problems.

In general, if you experience a mismatch, you should just recompile your
carrier or mezzanine driver with the same version of fmc-bus that you
are running as a kernel module.  If one of the structures has a new
field in there, the default of all-bits-as-zero will just work.  If your
running 'fmc.ko' is older than what the driver wants to use, you may
need to upgrade the core module, as the driver wants to use data fields
that are unknown to the core.  We foresee to never introduce backward
incompatibilities, so that step should be safe as well.

If your working place uses several drivers based on fmc-bus and you
feature non-trivial dependencies, the safest best is using the most
recent fmc-bus release you can find, and recompile all the rest against
that version.  Again, please remember that version changes are not
frequent events.


To help clarifying how the versions is used, here is an example.

Pretty often, the mezzanine drivers need to register a new device of
some kind: a network card, an input device, or something else.  Several
mezzanines may be installed in the same host computer, and we need a way
to tell which is which: using sequential numbering is not reliable.

When we moved the fine-delay driver from being a SPEC-only driver to a
mezzanine fmc-bus driver, we found we needed an identifier to register
our top-level device (here, a ZIO driver).  The right solution to this
problems is for carrier drivers to already spell out an identifier in
the 'fmc_device' structure, so mezzanine drivers can use it.  This
required a version change.

Drivers designed to work with an older versions of fmc-bus can just be
recompiled: they won't exploit the new feature but they will work
nonetheless.  In this case, an old carrier driver will unknowingly
register 0 as its identifier (and mezzanine drivers use "0" to trigger
sequential enumeration).  An old mezzanine driver will not use the
carrier-provided identifier and will use whatever technique it used
under the older fmc-bus version.

While writing the fine-delay driver, we chose to use some special
SPEC-dependent code to extract the identifier, until the core offered
the new field.  Carrier-specific code is explained in *note Suggestions
for Mezzanine Drivers::.

9.2 Git Submodule Crash Course
==============================

The mechanism we suggest to manage dependencies is git submodule.  It is
already in place in the existing FMC drivers, but using it requires some
knowledge.  This is some basic information about submodules, assuming
you are already accustomed to simple git repositories.  (Tutorials
exists, but they are usually too detailed as a quick start).

If you are just a user of fmc-bus, not developing drivers, you can jump
to the next section.

Quick Summary
-------------

To make a long story short: a git repository can know that it uses a
checkout of another git repository, at a specific version.  The
submodule is checked out using a commit identifier, so it must be
considered read-only by the supermodule (the top-level project).

If you checked out a specific branch of the submodule, the supermodule
will record the specific commit at the time of the checkout, not the
branch it was using: branches are not followed automatically nor even
recorded in the supermodule.

Supermodules and Submodules
---------------------------

Then, here's the longer story.  When you clone the supermodule, the
submodules won't be automatically cloned (unless you use 'git clone
--recursive', that nobody does by default and is not generally good for
fmc-bus drivers).  To check out the submodules you need to 'git
submodule init; git submodule update' or equivalent).

Whenever you modify the content of a checked-out submodule, it is
reported by 'git status', that considers the directory where the
submodule lives as a single file.  Your established use of 'git status'
will thus continue to work.  The messages you may get (if you changed
the submodule) are either 'modified content' or 'untracked content',
like this (in the fine-delay mezzanine driver):

        morgana$ echo > fmc-bus/new-file
        morgana$ echo >> spec-sw//Makefile
        morgana$ git status
        [...]
        #       modified:   fmc-bus (untracked content)
        #       modified:   spec-sw (modified content)

If you wonder where does git store the checkout information for the
submodule, it lives in the "tree object" (i.e.  directory) where the
submodule is checked-out.

        morgana$ git ls-tree  HEAD | grep commit
        160000 commit dc30ce998628084c4402291e6052c7feb77be424  fmc-bus
        160000 commit 35480a19bc562ebfa76f4663d41d026ff1b9df29  spec-sw

If you want a different commit in the submodule (for example, you fixed
a bug in it), this is tracked by a commit in the supermodule, because
the hosting tree must be modified to record the new submodule commit.
It is correct, in general, to have a commit in the supermodule that just
updates the commit identifier of the submodule.  Sometimes with,
however, you commit the new submodule's version together with changes in
the supermodule that use it (like they were parts of the same package).

How to Commit to a Submodule
----------------------------

There are two situations where you need to touch a submodule: either you
find a problem while working in the supermodule or you want to use a
newer version of the submodule.

In the latter case: the submodule has new useful commits and we want to
use them in the supermodule: just check-out the new upstream commit in
the submodule and then commit in the supermodule.

The former case is trickier: since the submodule is a read-only checkout
of another repository, normal users are not expected to commit to a
project from the place where it is just a submodule.  However, it
sometimes happen when your multi-repository project is still young, like
fmc-bus as of 2012.

The directory of the submodule looks like a normal git clone, so you can
work in it in your usual way.  However, the submodule is always
referencing a read-only upstream repository (because everybody should be
able to get the code), so you must add a new "remote" git reference,
like this:

        morgana$ git remote -v
        origin  git://ohwr.org/fmc-projects/fmc-bus.git (fetch)
        origin  git://ohwr.org/fmc-projects/fmc-bus.git (push)
        morgana$ git remote add work $HOME/ohwr/fmc-bus
        morgana$ git remote -v
        origin  git://ohwr.org/fmc-projects/fmc-bus.git (fetch)
        origin  git://ohwr.org/fmc-projects/fmc-bus.git (push)
        work    /home/rubini/ohwr/fmc-bus (fetch)
        work    /home/rubini/ohwr/fmc-bus (push)

Now you can work normally and push to work (if you don't push, you may
loose the commit when working in the supermodule, which doesn't consider
the submodule a working directory).  If you end up using the new commit
in the supermodule, you must commit the supermodule too as described
above.  Finally, you must ensure you push upstream the submodule before
you push the supermodule that uses the new commit.

9.3 Git Submodules and "make install"
=====================================

In the context of fmc-bus drivers a mezzanine driver relies on the core
fmc.ko and the carrier driver.  The user who downloads the mezzanine
driver would like to "make && make install" and find everything
installed.

While this approach is easily achieved by arranging for make install to
install the submodules as well, this will introduce problems for more
advanced users, who are using different mezzanines in the same host
system.  This is going to be pretty common, because you can drive FMC
mezzanines with Etherbone, so drivers in a single host won't be limited
to the few PCI or VME slots of your desktop computer or crate.

The suggested behaviour for an FMC driver package is using submodules
for the drivers it depends on.  However, it should only install its own
kernel modules.  If everything were installed automatically, we risked
overwriting already installed modules with a different version, and we
can't know if the user really wanted that - those other modules may be
already installed and verified as part of another set of drivers.

The suggested 'make install' rule, thus, only installs the current
package (the mezzanine or carrier driver the user checked out) and
prints this final line, explained by accompanying documentation.

        WARNING: Consider "make prereq_install"

If a user is running a host with this mezzanine alone, then installing
the prerequisites is a safe move.  In this case the warning message
should not appear if you make install again after you did 'make
prereq_install' once.  That's because technical users are expected to
sometimes edit and reinstall the specific driver, but not the ones it
depends on - such drivers are considered stable infrastructure, like the
kernel or the compiler.

9.4 Suggestions for Mezzanine Drivers
=====================================

A mezzanine driver should include fmc-bus and the relevant carrier
drivers as submodules, in order to easily access the needed headers.  A
mezzanine driver should compile the submodules as well, in order to find
the external symbols and not report scaring warnings at compile time.

While in a truly-portable mezzanine driver you won't need to access
headers of the carrier drivers, it's still good practice to carry them
as submodules, so the user can just 'make prereq_install' and have
everything that's needed to run the hardware item.

The examples shown here are taken from fine-delay, the first mezzanine
driver that uses this multi-level approach.  That specific driver also
uses ZIO as a submodule, but this has been stripped from the examples as
not relevant to fmc-bus.  The device driver itself lives in the kernel
subdirectory, because the package also features tools and lib.  This is
expected to be common, so all code shown below uses this convention;
adapting to a driver without subdirectories is expected to be easy.

First of all, the toplevel 'Makefile' should have all the standard
targets, and run all of them in each subdirectory.  Moreover, since most
users don't run "'git submodule init'" or similar commands, it should be
done automatically the first time to avoid errors that may be difficult
to diagnose.

These are the relevant lines of the 'Makefile':

       .PHONY: gitmodules prereq prereq_install prereq_install_warn
       all clean modules install modules_install: gitmodules
               [...]

       gitmodules:
               @test -d fmc-bus/doc || echo "Checking out submodules"
               @test -d fmc-bus/doc || git submodule update --init

       # The user can override, using environment variables, all these three:
       FMC_BUS ?= fmc-bus
       SPEC_SW ?= spec-sw
       SUBMOD = $(FMC_BUS) $(SPEC_SW)

       prereq:
               for d in $(SUBMOD); do $(MAKE) -C $$d || exit 1; done

       prereq_install_warn:
               @test -f .prereq_installed || \
                       echo -e "\n\n\tWARNING: Consider \"make prereq_install\"\n"

       prereq_install:
               for d in $(SUBMOD); do $(MAKE) -C $$d modules_install || exit 1; done
               touch .prereq_installed

The snippet above implements the following:

   * It checks out submodules if they are not already checked out.

   * It allows the user to override the submodules and use custom
     versions.  by setting environment variables ('FMC_BUS' and
     'SPEC_SW').

   * It implements the warning at installation time.

   * It has 'prereq_install', which stops the warning above.

The driver-specific makefile (here "'kernel/Makefile'") must take care
of finding the headers at compile time and the external symbols at link
time.  This is implemented using the standard Kbuild variables.  Again,
environment variable can be used to override the submodules with custom
clones of the repositories:

       LINUX ?= /lib/modules/$(shell uname -r)/build
       SPEC_SW ?= $(M)/../spec-sw
       FMC_BUS ?= $(M)/../fmc-bus

       KBUILD_EXTRA_SYMBOLS := \
               $(FMC_BUS)/kernel/Module.symvers


       ccflags-y = \
               -I$(SPEC_SW)/kernel \
               -I$(SPEC_SW)/kernel/include \
               -I$(FMC_BUS)/kernel/include \
               -I$M

If the mezzanine driver is truly portable (which fine-delay is not, as I
write this), you won't need to point 'ccflags-y' to '$(SPEC_SW)', which
is only compiled and installed for the user's convenience.

On the other hand, your mezzanine driver might also need to point
'KBUILD_EXTRA_SYMBOLS' to the carrier driver, whereas the example above
only uses symbols from fmc-bus.

9.5 Suggestions for Carrier Drivers
===================================

A carrier driver should include fmc-bus as a submodule, in order to
easily access the relevant header files.  It should also compile the
submodule, to access the symbols exported by 'fmc.ko' and avoid
unpleasant compilation warnings.

However, the repository of a carrier driver may be used as a submodule
of a mezzanine driver, which in turn already has fmc-bus as a submodule
(see *note Suggestions for Mezzanine Drivers::).  In general you don't
want fmc-bus to be replicated several times at different directory
levels if you clone a mezzanine driver.  If, by chance, the user of the
mezzanine driver changes something in the fmc-bus, nobody can easily say
which one is the one being re-installed.

For this reason, the carrier driver should not checkout the submodule if
both the carrier itself and the fmc-bus package are submodules of
another driver.  Actually, this is also why the 'gitmodules:' rule
suggested above for the mezzanine driver doesn't use '--recursive'.

The following example is taken from 'spec-sw', the first carrier driver
we developed for this bus abstraction:

       FMC_DRV ?= $(shell ./check-fmc-bus)
       export FMC_DRV

       DIRS = $(FMC_DRV) kernel tools

       all clean modules install modules_install:
               for d in $(DIRS); do $(MAKE) -C $$d $ || exit 1; done

As you see, the variable 'FMC_BUS' is conditionally assigned, so the
user can override it from the environment as usual.  Moreover, the
default value is returned by a local shell script.

The script 'check-fmc-bus' verifies whether both this directory and
'fmc-bus' are submodules of some other git repository living in the
parent directory.  If this is the case, 'FMC_BUS' is taken from the
parent directory, otherwise it is taken from the local subdirectory (the
submodule).

9.6 Work in Progress
====================

Unfortunately, these procedures still miss some bits, that will be dealt
with as the system increases in complexity and number of users.

For example, the "gitmodules" target is missing from carrier drivers, so
a developer working directly on the carrier code should be aware of
submodules, and run 'git submodule update --init' by hand after a fresh
checkout.  This is not yet performed by default because I need to
differentiate when the carrier driver is the main checkout from when it
is a submodule.

As the last bits are fixed I'll update this chapter, and I'll remove
this section when everything is verified by a larger user base.

10 FMC Identification
*********************

The FMC standard requires every compliant mezzanine to carry
identification information in an I2C EEPROM.  The information must be
laid out according to the "IPMI Platform Management FRU Information",
where IPMI is a lie I'd better not expand, and FRU means "Field
Replaceable Unit".

The FRU information is an intricate unreadable binary blob that must
live at offset 0 of the EEPROM, and typically extends for a few hundred
bytes.  The standard allows the application to use all the remaining
storage area of the EEPROM as it wants.

This chapter explains how to create your own EEPROM image and how to
write it in your mezzanine, as well as how devices and drivers are
paired at run time.  EEPROM programming uses tools that are part of this
package and SDB (part of the fpga-config-space package).

The first sections are only interesting for manufacturers who need to
write the EEPROM.  If you are just a software developer writing an FMC
device or driver, you may jump straight to *note SDB Support::.

10.1 Building the FRU Structure
===============================

If you want to know the internals of the FRU structure and despair, you
can retrieve the document from
<http://download.intel.com/design/servers/ipmi/FRU1011.pdf> .  The
standard is awful and difficult without reason, so we only support the
minimum mandatory subset - we create a simple structure and parse it
back at run time, but we are not able to either generate or parse more
arcane features like non-english languages and 6-bit text.  If you need
more items of the FRU standard for your boards, please submit patches.

This package includes the Python script that Matthieu Cattin wrote to
generate the FRU binary blob, based on an helper libipmi by Manohar
Vanga and Matthieu himself.  I changed the test script to receive
parameters from the command line or from the environment (the command
line takes precedence)

To make a long story short, in order to build a standard-compliant
binary file to be burned in your EEPROM, you need the following items:

        Environment    Opt     Official Name          Default
                                                      
---------------------------------------------------------------------
        FRU_VENDOR     -v      "Board Manufacturer"   fmc-example
                                                      
        FRU_NAME       -n      "Board Product Name"   mezzanine
                                                      
        FRU_SERIAL     -s      'Board Serial          0001
                               Number"                
        FRU_PART       -p      "Board Part Number"    sample-part
                                                      
        FRU_OUTPUT     -o      not applicable         /dev/stdout
                                                      

The "Official Name" above is what you find in the FRU official
documentation, chapter 11, page 7 ("Board Info Area Format").  The
output option is used to save the generated binary to a specific file
name instead of stdout.

You can pass the items to the FRU generator either in the environment or
on the command line.  This package has currently no support for
specifying power consumption or such stuff, but I plan to add it as soon
as I find some time for that.

FIXME: consumption etc for FRU are here or in PTS?

The following example creates a binary image for a specific board:

        ./tools/fru-generator -v CERN -n FmcAdc100m14b4cha \
               -s HCCFFIA___-CR000003 -p EDA-02063-V5-0 > eeprom.bin

The following example shows a script that builds several binary EEPROM
images for a series of boards, changing the serial number for each of
them.  The script uses a mix of environment variables and command line
options, and uses the same string patterns shown above.

        #!/bin/sh

        export FRU_VENDOR="CERN"
        export FRU_NAME="FmcAdc100m14b4cha"
        export FRU_PART="EDA-02063-V5-0"

        serial="HCCFFIA___-CR"

        for number in $(seq 1 50); do
           # build number-string "ns"
           ns="$(printf %06d $number)"
           ./fru-generator -s "${serial}${ns}" > eeprom-${ns}.bin
        done

10.2 Using SDB-FS in the EEPROM
===============================

If you want to use SDB as a filesystem in the EEPROM device within the
mezzanine, you should create one such filesystem using gensdbfs, from
the fpga-config-space package on OHWR.

By using an SBD filesystem you can cluster several files in a single
EEPROM, so both the host system and a soft-core running in the FPGA (if
any) can access extra production-time information.

We chose to use SDB as a storage filesystem because the format is very
simple, and both the host system and the soft-core will likely already
include support code for such format.  The SDB library offered by the
fpga-config-space is less than 1kB under LM32, so it proves quite up to
the task.

The SDB entry point (which acts as a directory listing) cannot live at
offset zero in the flash device, because the FRU information must live
there.  To avoid wasting precious storage space while still allowing for
more-than-minimal FRU structures, the fmc.ko will look for the SDB
record at address 256, 512 and 1024.

In order to generate the complete EEPROM image you'll need a
configuration file for gensdbfs: you tell the program where to place the
sdb entry point, and you must force the FRU data file to be placed at
the beginning of the storage device.  If needed, you can also place
other files at a special offset (we sometimes do it for backward
compatibility with drivers we wrote before implementing SDB for flash
memory).

The directory tools/sdbfs of this package includes a well-commented
example that you may want to use as a starting point (the comments are
in the file called --SDB-CONFIG--).  Reading documentation for gensdbfs
is a suggested first step anyways.

This package (generic FMC bus support) only accesses two files in the
EEPROM: the FRU information, at offset zero, with a suggested filename
of IPMI-FRU and the short name for the mezzanine, in a file called name.
The IPMI-FRU name is not mandatory, but a strongly suggested choice; the
name filename is mandatory, because this is the preferred short name
used by the FMC core.  For example, a name of "fdelay" may supplement a
Product Name like "FmcDelay1ns4cha" - exactly as demonstrated in
'tools/sdbfs'.

Note: SDB access to flash memory is not yet supported, so the short name
currently in use is just the "Product Name" FRU string.

The example in tools/sdbfs includes an extra file, that is needed by the
fine-delay driver, and must live at a known address of 0x1800.  By
running gensdbfs on that directory you can output your binary EEPROM
image (here below spusa$ is the shell prompt):

        spusa$ ../fru-generator -v CERN -n FmcDelay1ns4cha -s proto-0 \
                      -p EDA-02267-V3 > IPMI-FRU
        spusa$ ls -l
        total 16
        -rw-rw-r-- 1 rubini staff 975 Nov 19 18:08 --SDB-CONFIG--
        -rw-rw-r-- 1 rubini staff 216 Nov 19 18:13 IPMI-FRU
        -rw-rw-r-- 1 rubini staff  11 Nov 19 18:04 fd-calib
        -rw-rw-r-- 1 rubini staff   7 Nov 19 18:04 name
        spusa$ sudo gensdbfs . /lib/firmware/fdelay-eeprom.bin
        spusa$ sdb-read -l -e 0x100 /lib/firmware/fdelay-eeprom.bin
        /home/rubini/wip/sdbfs/userspace/sdb-read: listing format is to be defined
        46696c6544617461:2e202020  00000100-000018ff .
        46696c6544617461:6e616d65  00000200-00000206 name
        46696c6544617461:66642d63  00001800-000018ff fd-calib
        46696c6544617461:49504d49  00000000-000000d7 IPMI-FRU
        spusa$ ../fru-dump /lib/firmware/fdelay-eeprom.bin
        /lib/firmware/fdelay-eeprom.bin: manufacturer: CERN
        /lib/firmware/fdelay-eeprom.bin: product-name: FmcDelay1ns4cha
        /lib/firmware/fdelay-eeprom.bin: serial-number: proto-0
        /lib/firmware/fdelay-eeprom.bin: part-number: EDA-02267-V3

As expected, the output file is both a proper sdbfs object and an IPMI
FRU information blob.  The fd-calib file lives at offset 0x1800 and is
over-allocated to 256 bytes, according to the configuration file for
gensdbfs.

10.3 Accessing the EEPROM
=========================

The bus creates a sysfs binary file called eeprom for each mezzanine it
knows about:

        spusa.root# cd /sys/bus/fmc/devices; ls -l */eeprom
        -r--r--r-- 1 root root 8192 Feb 21 12:30 FmcAdc100m14b4cha-0800/eeprom
        -r--r--r-- 1 root root 8192 Feb 21 12:30 FmcDelay1ns4cha-0200/eeprom
        -r--r--r-- 1 root root 8192 Feb 21 12:30 FmcDio5cha-0400/eeprom

Everybody can read the files and the superuser can also modify it, but
the operation may on the carrier driver, if the carrier is unable to
access the I2C bus.  For example, the spec driver can access the bus
only with its golden gateware: after a mezzanine driver reprogrammed the
FPGA with a custom circuit, the carrier is unable to access the EEPROM
and returns ENOTSUPP.

An alternative way to write the EEPROM is the mezzanine driver
fmc-write-eeprom (See *note fmc-write-eeprom::), but the procedure is
more complex.

11 SDB Support
**************

The fmc.ko bus driver exports a few functions to help drivers taking
advantage of the SDB information that may be present in your own FPGA
memory image.

The module exports the following functions, in the special header
<linux/fmc-sdb.h>.  The linux/ prefix in the name is there because we
plan to submit it upstream in the future, and don't want to force
changes on our drivers if that happens.

         int fmc_scan_sdb_tree(struct fmc_device *fmc, unsigned long address);
         void fmc_show_sdb_tree(struct fmc_device *fmc);
         signed long fmc_find_sdb_device(struct sdb_array *tree, uint64_t vendor,
                                         uint32_t device, unsigned long *sz);
         int fmc_free_sdb_tree(struct fmc_device *fmc);

To be completed.

12 Portability
**************

This package should be portable.  However I didn't test it on a wide
variety of system.  I used it on a 32-bit x86 host, running version 3.4
of the kernel.

The package is upstream since version 3.11, and I'm submitting all local
changes to the kernel as development continues.  As of this release a
few commits are missing from the official kernel, but any ohwr package
that relies on fmc-bus will most likely work with the official kernel
subsystem as well.

I verified that this package builds without any warning from version
2.6.35 up to 3.13, on a 32-bit x86 processor.  On 2.6.34 and earlier it
fails for various reasons.

Up to May 2013 each release included a backport branch that worked back
to kernel 2.6.24, which we had in production at the time.  If you run a
kernel older than 2.6.35, please consider

