Implementation of of FTD2XX-based AVR programmer in avrdude

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Implementation of of FTD2XX-based AVR programmer in avrdude

Johnathan Corgan
I've spent the day hacking together an experimental implementation of
serial programming via the FTDI FT232BM USB-to-serial converter bit bang
mode.  Mostly a success, so far.

After familiarizing myself with the programmer_t abstraction and seeing
how it is implemented in par.c and other files, I was able to create a
new programmer type ('ftbb') and make a new programmer entry in
avrdude.conf that uses it.  I did consider first trying to fit into
Michael Holtz' bit bang structure but there were more differences than
similarities, so I went the new interface route.

At the lowest level, there is a routine called ftbb_txrx() which sends
and receives 8 bits over the bit bang bus of the FTD2XX.

pgm->cmd builds on this and sends four bytes of a command and records
the four bytes of the response.

The other routines for open, close, enable, disable, initialize, erase,
etc., are straightforward, making the right calls to the FTDI DLL.

I am able to use the terminal mode successfully with an ATMega8 for
everything I've tried so far.  All the memory space read commands from
the command line are working too.

There does seem to be a subtle timing bug as I can do a chip erase from
the terminal but not from the command line--the SPI bus falls out of
sync and doesn't respond any further.  I do have the usleep() call with
the erase delay value in there, but maybe that's not where the problem lies.

However--this implementation is *extremely slow*.  It accomplishes about
  four commands per second, so reading 16 bytes of eeprom takes, for
example, 4 seconds.  This is due ultimately to requiring a USB packet
per SCLK transition, with a 200ms or so fixed timeout inside the FTD2XX
device driver DLL.  (I know, Joerg warned about this--but stay with me.)

As I outlined in a prior email, this situation happens because one has
to read back the status of the bit bang bus pins after SCLK goes low,
shifting the FTDI driver from transmitting to receiving for each bit.
This creates a ping-pong effect that kills the performance.

The much, much faster technique is to batch together bit bang bus sends
into a larger block (again, see previous email), and let the FTDI chip
clock these out over the bit bang bus at high speed.  For a single
command, one would convert 32 bits into 64 bytes that would get sent as
a single USB packet.

This mode of operation, though, prevents any reads of the bit bang bus,
and thus the response from AVR is lost.  There are many commands,
however, for which this should be ok.  A stream of memory space writes
(flash or eeprom) could go quickly using this technique.

Unfortunately, avrdude does not internally distinguish between commands
which require a response and ones that don't, so I'm forced to do the
very slow method for everything.  If there were a way in pgm->cmd() to
know that the response would be ignored, I could send the 32 bits all at
once in the fast way outlined above.

I can see that there is an opportunity to implement a paged write
routine that could batch these up and make it work that way, and other
batching techniques could be implemented in pgm->cmd() if I knew how to
figure out an appropriate time to stop batching and send out the buffer.
  But again this would require knowing which commands need a response
and which don't.

Suggestions?

A patch is forthcoming, but I'm a little brain fried right now to do it.
  The implementation is a quite fragile--Cygwin only, needs the FTDI
header file and .lib DLL import file in the current directory, and has
no conditionalization in the Makefile.am or source code files.  It also
ignores the port specification (just uses FTD2XX device #0), ignores the
pin settings in avrdude.conf, and assumes things are wired exactly as I
have them on my board.

As is, I could use this for my AVR project, so my itch is mostly
scratched.  But having gone this far, I'd sure like to see how it can be
integrated nicely and optimized for speed.

-Johnathan



_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Uwe Bonnes
Johnathan Corgan <[hidden email]> wrote:
> I've spent the day hacking together an experimental implementation of
> serial programming via the FTDI FT232BM USB-to-serial converter bit bang
> mode.  Mostly a success, so far.

> After familiarizing myself with the programmer_t abstraction and seeing
> how it is implemented in par.c and other files, I was able to create a
> new programmer type ('ftbb') and make a new programmer entry in
> avrdude.conf that uses it.  I did consider first trying to fit into
> Michael Holtz' bit bang structure but there were more differences than
> similarities, so I went the new interface route.

> At the lowest level, there is a routine called ftbb_txrx() which sends
> and receives 8 bits over the bit bang bus of the FTD2XX.

> pgm->cmd builds on this and sends four bytes of a command and records
> the four bytes of the response.

> The other routines for open, close, enable, disable, initialize, erase,
> etc., are straightforward, making the right calls to the FTDI DLL.

> I am able to use the terminal mode successfully with an ATMega8 for
> everything I've tried so far.  All the memory space read commands from
> the command line are working too.

> There does seem to be a subtle timing bug as I can do a chip erase from
> the terminal but not from the command line--the SPI bus falls out of
> sync and doesn't respond any further.  I do have the usleep() call with
> the erase delay value in there, but maybe that's not where the problem lies.

> However--this implementation is *extremely slow*.  It accomplishes about
>   four commands per second, so reading 16 bytes of eeprom takes, for
> example, 4 seconds.  This is due ultimately to requiring a USB packet
> per SCLK transition, with a 200ms or so fixed timeout inside the FTD2XX
> device driver DLL.  (I know, Joerg warned about this--but stay with me.)

Learn about the latency timer inside the FT2232. I will encrease speed
substancial. Also try to send big chunks to the FT2232. This will encrease
speed too.


Bye

--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------



_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Joerg Wunsch
As Uwe Bonnes wrote:

> Learn about the latency timer inside the FT2232. I will encrease
> speed substancial.

Keep in mind that this will also drastically increase the interrupt
load of the machine, so you might want to restrict decreasing the
latency timer to those situations where it's actually needed.
(1 ms latency timer => 1000 interrupts/s)

--
cheers, J"org               .-.-.   --... ...--   -.. .  DL8DTL

http://www.sax.de/~joerg/                        NIC: JW11-RIPE
Never trust an operating system you don't have sources for. ;-)


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Joerg Wunsch
In reply to this post by Johnathan Corgan
As Johnathan Corgan wrote:

> The other routines for open, close, enable, disable, initialize,
> erase, etc., are straightforward, making the right calls to the FTDI
> DLL.

Keep in mind that we'll need a Unix implementation as well...

--
cheers, J"org               .-.-.   --... ...--   -.. .  DL8DTL

http://www.sax.de/~joerg/                        NIC: JW11-RIPE
Never trust an operating system you don't have sources for. ;-)


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Uwe Bonnes
In reply to this post by Joerg Wunsch
Joerg Wunsch <[hidden email]> wrote:
> As Uwe Bonnes wrote:

> > Learn about the latency timer inside the FT2232. I will encrease
> > speed substancial.

> Keep in mind that this will also drastically increase the interrupt
> load of the machine, so you might want to restrict decreasing the
> latency timer to those situations where it's actually needed.
> (1 ms latency timer => 1000 interrupts/s)

If the host is a modern GHz PC, do you expect any real impact by 1000
Interrupts per second?  For an embedded host, I agree.
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------



_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Uwe Bonnes
In reply to this post by Joerg Wunsch
Joerg Wunsch <[hidden email]> wrote:
> As Johnathan Corgan wrote:

> > The other routines for open, close, enable, disable, initialize,
> > erase, etc., are straightforward, making the right calls to the FTDI
> > DLL.

> Keep in mind that we'll need a Unix implementation as well...

There is a UNIX verison of the D2XX library. However when I tried beginning
this year, I hit a hard wall with some problems with FT_GetBitMode.
support2@ftdichip agreed on that problem and promised to fix "in the next
release". The version on the web however is still from October 2004.

The problem was a showstopper andf I switched to the open source libftdi
( based on libusb). As libusb is available on Windows, things should somehow
be portable.  

I have adapted Anton Erasmus javr and Andrew Rogers xc3progs  to use an
FT2232. The code is in bad shape, using C++ elements. If somebody is
interessted in improving. I am willing to share.

--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------



_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Joerg Wunsch
In reply to this post by Uwe Bonnes
As Uwe Bonnes wrote:

> > Keep in mind that this will also drastically increase the interrupt
> > load of the machine, so you might want to restrict decreasing the
> > latency timer to those situations where it's actually needed.
> > (1 ms latency timer => 1000 interrupts/s)

> If the host is a modern GHz PC, do you expect any real impact by 1000
> Interrupts per second?  For an embedded host, I agree.

Don't make assumptions.  AVRDUDE is not unlikely to be used in lab
environments with older PCs, like a 400 MHz CPU.  Just that your
current desktop machine wouldn't have much problems with it is no
justification to waste resources if you can do better.

Sure, it could be argued that USB with its missing option for a device
to issue a real interrupt is just crap by design, but that's another
matter.
--
cheers, J"org               .-.-.   --... ...--   -.. .  DL8DTL

http://www.sax.de/~joerg/                        NIC: JW11-RIPE
Never trust an operating system you don't have sources for. ;-)


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Johnathan Corgan
In reply to this post by Uwe Bonnes
Uwe Bonnes wrote:

> Learn about the latency timer inside the FT2232. I will encrease speed
> substancial. Also try to send big chunks to the FT2232. This will encrease
> speed too.

I'm using the FT232BM.  The FTD2XX library supplies a FT_SetLatencyTimer
function.  I've set this to the minimum (1 ms), it doesn't seem to have
any effect.  Have not spent any time debugging yet.

-Johnathan


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Johnathan Corgan
In reply to this post by Joerg Wunsch
Joerg Wunsch wrote:

> Keep in mind that we'll need a Unix implementation as well...

Sure.  I'm not yet familiar with the build system in place for
distinguishing between platforms.  I see that there is a separate
directory for Windows and a WIN32NATIVE symbol defined, so I'll figure
out how to use these for the Cygwin version.

I haven't used the Linux drivers for FTDI yet, I don't know how similar
they are to the Windows ones.  If they correspond one for one then it
should be easy.

Also, I'm not particularly knowledgeable about autoconf/automake.  For
the Win32 version, one must link in FTD2XX.lib.  I did this with a line:

avrdude_LDADD = FTD2XX.lib

in the Makefile.am file.  But this needs to put the .lib file in the
current directory.  I'd like instead to grab the copy in the installed
FTDI driver directory.

Under Linux I'm sure there will be a similar .so or .a to link to.
Again, I don't know how to conditionalize things in Makefile.am to do
the right thing when the build is on Linux vs. Windows.

Since my AVR project will ultimately be Windows based I've set up using
Cygwin for now.  In a few days I may take a crack at Linux.

I leave on a business trip tonight until Thursday, so I probably won't
get to post the patch until then.

-Johnathan




_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Johnathan Corgan
In reply to this post by Uwe Bonnes
Uwe Bonnes wrote:

> The problem was a showstopper andf I switched to the open source libftdi
> ( based on libusb). As libusb is available on Windows, things should somehow
> be portable.  

I've glanced through the ftdi.c code from Intra2net website.  It looks
platform independent, depending only on libusb.  Has anyone compiled
this as a Win32 DLL and used it with libusb on Windows?

This would be the ideal situation, allowing identical function call
names between platforms and a completely open-source solution.

-Johnathan


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Uwe Bonnes
In reply to this post by Johnathan Corgan
Johnathan Corgan <[hidden email]> wrote:
> Uwe Bonnes wrote:

> > Learn about the latency timer inside the FT2232. I will encrease speed
> > substancial. Also try to send big chunks to the FT2232. This will encrease
> > speed too.

> I'm using the FT232BM.  The FTD2XX library supplies a FT_SetLatencyTimer
> function.  I've set this to the minimum (1 ms), it doesn't seem to have
> any effect.  Have not spent any time debugging yet.

Try setting it to 2 ms. For me, 1 ms had no effect.

--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------



_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Johnathan Corgan
Uwe Bonnes wrote:

> Try setting it to 2 ms. For me, 1 ms had no effect.

No effect.  Nor did 10 ms.

I'm not certain this parameter is the problem anyway.  My understanding
is the latency timer governs how long to wait for further user writes
before sending an incomplete buffer.

In this case the delay seems to be a turn-around issue.  I'm sending a
byte with SCLK low using FT_Write, then reading a byte with
FT_GetBitMode for the pin status, then sending a byte with SCLK high,
etc. Maybe it's the case that there is a receive latency timer in the
hardware that isn't settable, and that's where the delay is.

Anyway, that's probably enough FTDI specific email traffic for the list.
  When I get back from my trip I'll dive into this with a fresh mind,
and set up a support call with FTDI if needed.

I may hack in a flag into avrpart.h that indicates whether each command
needs a result, and take advantage of that in pgm->cmd() to batch write
the 32 bits and fake a response value.

-Johnathan


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Uwe Bonnes
Johnathan Corgan <[hidden email]> wrote:
> Uwe Bonnes wrote:

> > Try setting it to 2 ms. For me, 1 ms had no effect.

> No effect.  Nor did 10 ms.

For me (and using libftdi) there was a tremendous effect.
--
Uwe Bonnes                [hidden email]

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------



_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Matthias Weißer
Uwe Bonnes schrieb:
>> > Try setting it to 2 ms. For me, 1 ms had no effect.
>
>> No effect.  Nor did 10 ms.
>
> For me (and using libftdi) there was a tremendous effect.

Me too. If I set the timeout from 16ms to 1ms within the VCP driver
under Windows my USBisp is up to three times faster depending on a the
operation I am executing.

--
Matthias Wei?er
[hidden email]
http://www.matwei.de


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Johnathan Corgan
Matthias Wei?er wrote:
> Uwe Bonnes schrieb:
>> For me (and using libftdi) there was a tremendous effect.
>
> Me too. If I set the timeout from 16ms to 1ms within the VCP driver
> under Windows my USBisp is up to three times faster depending on a the
> operation I am executing.

Ok.  Will let the list know when I make more progress.

-Johnathan




_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Joerg Wunsch
In reply to this post by Matthias Weißer
As Matthias Weißer wrote:

> >For me (and using libftdi) there was a tremendous effect.

> Me too. If I set the timeout from 16ms to 1ms within the VCP driver
> under Windows my USBisp is up to three times faster depending on a
> the operation I am executing.

For me, it's also been quite an improvement -- but that's with
standard serial mode.  Are you sure the latency timer also affects
bit-bang mode at all?

--
cheers, J"org               .-.-.   --... ...--   -.. .  DL8DTL

http://www.sax.de/~joerg/                        NIC: JW11-RIPE
Never trust an operating system you don't have sources for. ;-)


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev
Reply | Threaded
Open this post in threaded view
|

Re: Implementation of of FTD2XX-based AVR programmer in avrdude

Matthias Weißer
Joerg Wunsch schrieb:

>> Me too. If I set the timeout from 16ms to 1ms within the VCP driver
>> under Windows my USBisp is up to three times faster depending on a
>> the operation I am executing.
>
> For me, it's also been quite an improvement -- but that's with
> standard serial mode.  Are you sure the latency timer also affects
> bit-bang mode at all?

Absolut not. I see I didn't had the complete thread in focus. So if the
discussion was about bit bang mode I didn't hava any experience.

--
Matthias Wei?er
[hidden email]
http://www.matwei.de


_______________________________________________
avrdude-dev mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/avrdude-dev