Implement OTA Firmware upgrade using CC3000

[satishgn]

[satishgn]

Before implementing the OTA/WLAN based DFU, we have the following things in place:
1)Integration of working SPI Flash driver with DFU codebase.
2)Integration of working CC3000/WLAN driver with DFU codebase.
3)Application Backup functionality from Internal Flash to External Flash.
4)Application Restore functionality from External flash to Internal Flash.

Note:
Application "marvin" address in Internal Flash has been fixed at 0x08007000
and External Flash Backup/Restore address has been fixed at 0x00007000

Here is the next plan and TODO items:

When the marvin application is running, it will wait for command/message from the API server for new Firmware availability.

As soon as the "Upgrade_Firmware" message is received, the core will reset itself into DFU mode.

Now there are few ways how we can proceed with OTA firmware upgrade in DFU codebase:

1)Implement a basic TFTP server on core. Use a TFTP client on PC/Server side to initiate the firmware file transfer for writing on to the Internal Flash memory. This should be very easy to implement.
2)Implement a basic HTTP server on core. Upload Firmware file using HTML forms. To send raw binary data, the HTML POST method is used instead of the GET.
3)Implement TCP socket communication for firmware upgrade. Here we need a server side code which sends blocks of firmware upgrade code along with CRC checksum and core perform a CRC check before writing onto the Flash.

Zac, If you wish to take #3 approach, I need your help in getting the server side code implemented.
Let me know if you have other plan on OTA-DFU approach.

[zsup]

Good suggestions for OTA firmware update. I think the biggest concern is that whatever solution we choose in the DFU codebase must be relatively foolproof, since it will be difficult to update or change once the Cores are in peoples' hands. Which of the three do you think would be least likely to requires changes or updates further down the road?

Zach Supalla
312.953.3413

On Thursday, May 23, 2013 at 11:42 AM, Satish Nair wrote:

Before implementing the OTA/WLAN based DFU, we have the following things in place:
1)Integration of working SPI Flash driver with DFU codebase.
2)Integration of working CC3000/WLAN driver with DFU codebase.
3)Application Backup functionality from Internal Flash to External Flash.
4)Application Restore functionality from External flash to Internal Flash.
Note:
Application "marvin" address in Internal Flash has been fixed at 0x08007000
and External Flash Backup/Restore address has been fixed at 0x00007000
Here is the next plan and TODO items:
When the marvin application is running, it will wait for command/message from the API server for new Firmware availability.
As soon as the "Upgrade_Firmware" message is received, the core will reset itself into DFU mode.
Now there are few ways how we can proceed with OTA firmware upgrade in DFU codebase:
1)Implement a basic TFTP server on core. Use a TFTP client on PC/Server side to initiate the firmware file transfer for writing on to the Internal Flash memory. This should be very easy to implement.
2)Implement a basic HTTP server on core. Upload Firmware file using HTML forms. To send raw binary data, the HTML POST method is used instead of the GET.
3)Implement TCP socket communication for firmware upgrade. Here we need a server side code which sends blocks of firmware upgrade code along with CRC checksum and core perform a CRC check before writing onto the Flash.
Zac, If you wish to take #3 (#3) approach, I need your help in getting the server side code implemented.
Let me know if you have other plan on OTA-DFU approach.

—
Reply to this email directly or view it on GitHub (#3 (comment)).

[towynlin]

We should use approach 3, sending the data with appropriate checksums over the existing socket. Understood: I and/or someone else we hire will work with you to write the server-side code.

@satishgn Some things to keep in mind:

1. We will soon switch to protocol buffers for the communication between the device and server.
2. Will need to move to C++ soon.
3. We need to move toward test-driven development. Can you request expense reimbursement through oDesk? If so (assuming you don't already have a copy), please buy a $25 eBook of Test Driven Development for Embedded C here: http://pragprog.com/book/jgade/test-driven-development-for-embedded-c and Spark will reimburse you for it. Read it as you have time, and start thinking about testing our existing code and writing tests firsts for new code.

[satishgn]

Sure, will try to get my hands on that eBook.

On Fri, May 24, 2013 at 2:27 AM, Zachary Crockett
notifications@github.comwrote:

We should use approach 3, sending the data with appropriate checksums over
the existing socket. Understood: I and/or someone else we hire will work
with you to write the server-side code.

@satishgn https://github.com/satishgn Some things to keep in mind:

1. We will soon switch to protocol buffers for the communication
   between the device and server.
2. Will need to move to C++ soon.
3. We need to move toward test-driven development. Can you request
   expense reimbursement through oDesk? If so (assuming you don't already have
   a copy), please buy a $25 eBook of Test Driven Development for Embedded C
   here:
   http://pragprog.com/book/jgade/test-driven-development-for-embedded-cand Spark will reimburse you for it. Read it as you have time, and start
   thinking about testing our existing code and writing tests firsts for new
   code.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18371167
.

[satishgn]

I have a suggestion on the partition of our Internal Flash structure.
Currently we have:
1)DFU code (USB:complete + OTA:under development)at the start of Flash at address 0x08000000 (28KB reserved)
2)Marvin code at address 0x08007000 (100KB available)

My suggestion is since our USB-DFU is relatively foolproof and reliable, we can keep it separate from OTA-DFU.
We can have the following:
1)DFU (USB) code at 0x08000000 (19KB reserved) and locked(non-writable)
2)System/User Flags at 0x08004C00 (1KB reserved). This is the alternative for the Backup registers/External Flash (as per Andy's feedback)
3)OTA-DFU code at 0x08005000 (28KB reserved)
4)Marvin code at 0x0800A000 (80KB available)

While shipping out core, we can flash the USB-DFU code and use it to flash 3 and 4 using dfu-util tool.
To enter USB-DFU mode, the button needs to be pressed at reset.
To enter OTA-DFU mode, the Server API will send a request to marvin application to reset itself to OTA upgrade.

Let me know your viewpoints on the above.

[andyw-lala]

I am not seeing the big picture. To run ota-dfu is it required for the
stm32 to read external flash, program internal flash with the ota-dfu code,
then write the ota-dfu payload to external flash ? Or does it execute
ota-dfu from ram ?

Don't forget that you will want 2 copies of anything downloadable (perhaps
you consider this done with a running copy in stm32 flash and an external
copy.

We must avoid the situation where a failed download results in losing a
previously working firmware.
On 24 May 2013 00:16, "Satish Nair" notifications@github.com wrote:

I have a suggestion on the partition of our Internal Flash structure.
Currently we have:
1)DFU code (USB:complete + OTA:under development)at the start of Flash at
address 0x08000000 (28KB reserved)
2)Marvin code at address 0x08007000 (100KB available)

My suggestion is since our USB-DFU is relatively foolproof and reliable,
we can keep it separate from OTA-DFU.
We can have the following:
1)DFU (USB) code at 0x08000000 (19KB reserved) and locked(non-writable)
2)System/User Flags at 0x08004C00 (1KB reserved). This is the alternative
for the Backup registers/External Flash (as per Andy's feedback)
3)OTA-DFU code at 0x08005000 (28KB reserved)
4)Marvin code at 0x0800A000 (80KB available)

While shipping out core, we can flash the USB-DFU code and use it to flash
3 and 4 using dfu-util tool.
To enter USB-DFU mode, the button needs to be pressed at reset.
To enter OTA-DFU mode, the Server API will send a request to marvin
application to reset itself to OTA upgrade.

Let me know your viewpoints on the above.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18386999
.

[satishgn]

All the executable code(usb-dfu, ota-dfu & marvin) will be organized in Internal flash.
The External Flash will just contain the backup copies of working marvin code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG programmer. The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu and marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed with the latest firmware.
In the event that the crc checksum of the latest firmware fails or other failed download conditions,
the backup marvin copy in External flash will be Restored/programmed back to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can anytime program both the Internal and External Flash via USB.

[andyw-lala]

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external. The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" notifications@github.com wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized in
Internal flash.
The External Flash will just contain the backup copies of working marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18392469
.

[satishgn]

Sounds good. During production, all the binaries can be combined together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala notifications@github.comwrote:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then
also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external. The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" notifications@github.com wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized in
Internal flash.
The External Flash will just contain the backup copies of working marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18395171
.

[andyw-lala]

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" notifications@github.com wrote:

Sounds good. During production, all the binaries can be combined together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala notifications@github.comwrote:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then
also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" notifications@github.com wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via USB.

Flags in Backup registers will be used for switching between ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18395464
.

[satishgn]

I don't see a need to program External Flash during production.
The External Flash will be erased and programmed with the back copy of the
marvin code when the OTA-DFU code is executed. The External Flash can also
be programmed over USB by the dfu-util tool when the USB-DFU code is
executed.

On Fri, May 24, 2013 at 3:55 PM, andyw-lala notifications@github.comwrote:

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" notifications@github.com wrote:

Sounds good. During production, all the binaries can be combined
together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala notifications@github.comwrote:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and then
also
loads a small programming into ram that is responsible for formatting
external flash and copying sections from stm32 flash out to external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" notifications@github.com wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via
USB.

Flags in Backup registers will be used for switching between ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user can
anytime program both the Internal and External Flash via USB.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395464>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18397302
.

[andyw-lala]

As long as all the possible failure modes have been carefully thought
through perhaps you do not. OTA requires very careful and methodical
planning.
On 24 May 2013 05:33, "Satish Nair" notifications@github.com wrote:

I don't see a need to program External Flash during production.
The External Flash will be erased and programmed with the back copy of the
marvin code when the OTA-DFU code is executed. The External Flash can also
be programmed over USB by the dfu-util tool when the USB-DFU code is
executed.

On Fri, May 24, 2013 at 3:55 PM, andyw-lala notifications@github.comwrote:

But since JTAG cannot write external flash, plan for a program to init
that, based on the internal flash contents. Best way to do that without
further flash hacking is to load a ram-based binary via JTAG.
Disclaimer: I've done this for msp430 but not cortex yet, but I
understand
it should be possible.
On 24 May 2013 04:38, "Satish Nair" notifications@github.com wrote:

Sounds good. During production, all the binaries can be combined
together
into a single file and programmed at one go.

On Fri, May 24, 2013 at 3:01 PM, andyw-lala notifications@github.comwrote:

OK I think I understand. I would recommend that the production
test/programming fixture blows a combo image into stm32 flash and
then
also
loads a small programming into ram that is responsible for
formatting
external flash and copying sections from stm32 flash out to
external.
The
combo image can contain the per unit personalisation data too.
On 24 May 2013 03:26, "Satish Nair" notifications@github.com
wrote:

All the executable code(usb-dfu, ota-dfu & marvin) will be
organized
in
Internal flash.
The External Flash will just contain the backup copies of working
marvin
code.

With a fresh core in hand,
spark will first program the usb-dfu application using a JTAG
programmer.
The dfu pages will be write-protected.
Then using dfu-util, ota-dfu is programmed followed by marvin via
USB.

Flags in Backup registers will be used for switching between
ota-dfu
and
marvin after System reset. This is a good use of Backup Registers.

When in ota-dfu mode, the first step is to create a backup copy of
marvin(application code) in External flash.
Then the marvin code in Internal flash will be erased and
programmed
with
the latest firmware.
In the event that the crc checksum of the latest firmware fails or
other
failed download conditions,
the backup marvin copy in External flash will be
Restored/programmed
back
to the Internal flash.

Also since the usb-dfu is failproof and reliable, the core user
can
anytime program both the Internal and External Flash via USB.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18392469>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395171>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18395464>
.

—
Reply to this email directly or view it on GitHub<
https://github.com/sprk/dfu/issues/3#issuecomment-18397302>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-18397605
.