Speeduino is now on Github Sponsors (Rather than Patreon): https://github.com/sponsors/noisymime
Any general discussion around the firmware, what is does, how it does it etc.
#47230
As the title suggest a post on sd card logging for speeduino.

I have been working on the sd-card logging functionality for some time now. But I hit a snag that is not easily worked around I think. It is not impossible but a lot more work to fix! So I thought let me share this with everyone. Use this post as a reference to know what you can expect. I have been trying this on an Ebay STM32 board. But the main problem/snag remains the same for all boards.

There are some good things also :-D. The following things are working in tests I have done..
  • Setting the Realt-time clock with Tuner studio to current computer time.
  • Using the Realt-time clock for timestamp of the file name
  • Using the real-time clock for file creation time and date.
  • Datalogging in CSV format of most currentstatus struct variables. (Partly working, with crippling limitations)
Libraries used.
  • FatFs from elm-chan.org/fsw/ff slightly adapted by ST for use with the STM32 MCU series
  • STM32duino STM32SD (that dependents on the library mention above)
  • STM32duino RTC (Used for time stamping and file creating time/date)
The FatFs works perfect in a separate sketch! But there are two problems when using it in the speeduino project.

1. I suspect the interrupts of speeduino create problems with the SDIO interface. Sdlogging works for a while but it flunks out after some random time. This can be solved and is more an issue with the STM32 and not in general.
2. It blows the rest of the real-timeliness of the board till the point is starts missing ignition pulses. (This is most likely a general sdcard/FatFs problem)

The second point is likely a sdcard interface limitations in combination with the FatFs library write function being blocking. It must be asynchronous or DMA based to work.

95% of the time a write of 512 bytes (secotr size) to the sdcard FAT file system takes 1.5 milliseconds. That is long but hopefully ok, I do not expect big problems with that. But when hitting some physical bound of the sdcard (In my case every ~265kb) the sdcard reports it is busy and the write function waits until sdcard is ready. This can take >100ms. During this time the main loop of speeduino would be blocked creating all sorts of problems.

The solution in my opinion is an asynchronous (or DMA) version of the FatFs file system library. Where a write to the file system that waits on the sdcard returns with a “waiting on sdcard” flag. So that the main loop can continue and writing the same info can be tried a next loop. This solution also needs a asynchronous Sdcard I/O layer that is not there yet.

Because sdcard logging function is not strictly needed with data logging using phones or other devices i stop working on it for the moment. Or until i have a very bright light bulb to fix it :D. (Or someone here on the forum knows a good solution for this of-course ).
#47231
I agree, there is less demand for SD logging, as other devices can be easily used for that purpose, such as phones. Still, it would be convenient to have a background full-time logging system that could catch and save unexpected events for analysis. Overwriting older log files as the card fills could allow this.

There have been multiple attempts at SD logging, without good success. @dazq had similar results years ago, solved by calling the data slowly so the SD could be written, and so a very slow capture rate. Let me ask a stupid question, to see if a solution is as simple as I hope. :roll:

It the problem that we are trying to write to SD directly, without a large buffer? TunerStudio and MegaLogViewer have no issues retrieving data smoothly for dashboard and logging at maximum rate, but they collect the data in a buffer (memory file), then write to a card (if desired) as a separate following step. Could it be that we need an SD module that retrieves data to a memory buffer (F-RAM chip?). That stored data is then written to SD as slowly or quickly as it wants, without interrupting Speeduino. I'm wondering if a multi-MB data buffer is one solution, or at least not trying to write directly to SD.Image
#47238
Not a full solution. The quality of the SD card may be in question as well.

My friend bought a camera to mount on his motorcycle for a European trip. It wouldn't record reliably, until he bought a quality SD card.

Customer service in the shop, explained cheap memory is slower.
#47243
PSIG wrote:
Tue Dec 22, 2020 9:45 pm
I agree, there is less demand for SD logging, as other devices can be easily used for that purpose, such as phones. Still, it would be convenient to have a background full-time logging system that could catch and save unexpected events for analysis. Overwriting older log files as the card fills could allow this.

There have been multiple attempts at SD logging, without good success. @dazq had similar results years ago, solved by calling the data slowly so the SD could be written, and so a very slow capture rate. Let me ask a stupid question, to see if a solution is as simple as I hope. :roll:

It the problem that we are trying to write to SD directly, without a large buffer? TunerStudio and MegaLogViewer have no issues retrieving data smoothly for dashboard and logging at maximum rate, but they collect the data in a buffer (memory file), then write to a card (if desired) as a separate following step. Could it be that we need an SD module that retrieves data to a memory buffer (F-RAM chip?). That stored data is then written to SD as slowly or quickly as it wants, without interrupting Speeduino. I'm wondering if a multi-MB data buffer is one solution, or at least not trying to write directly to SD.Image
Yes the buffering Was an issue with the 328p based devices that were being experimented with at the time.
A better MCU and more memory to buffer with would make for a much better hardware base.

One issue that does still remain is the need to close the file on the memory card before power off. If not closed the file becomes potentially unreadable.
#47258
This is just on the STM32F407 black boards that it is tested. The sdcard is on the SDIO port not the SPI port.

For the speeduino project: checkout this: https://github.com/Tjeerdie/speeduino/t ... DCARD_2021

1. Insert an sdcard formatted FAT16 or FAT32
2. Upload project to board
3. open TS project and open new ini, make sure hardware testing is enabled!
4. When ts connects, go to “Hardware testing”
5. Setup real-time clock
6. press “set MS3 time to NOW” this will setup the RTC of the STM32 to current computer time
7. Go to “Hardware testing”
8. Press “sdcard init test”. Indicators on the bottom should now show “sd in”
9. Press “sdcard open log file test”. Indicators on the bottom should now show “sd in” + “sd ready”
10. Press “sdcard close log file test” before “SD err” shows up! (can go wrong really fast)
11. put the sdcard in your computer. There must be an csv file with log
13. You can open a terminal on the hardware serial port 1 to see some debug messages.

It is not working because something is making the sdcard behaviour bad. Probably solvable but even if it is solved the long blocking delay on the 256kb boundries on the sdcard are not solved. So I did not look further into this.

For the sdcard logging test sketch to show the long blocking time. Check out the following:
https://github.com/Tjeerdie/sdcard-test

1. Insert an sdcard with FAT16 or FAT32 formatting
2. Upload project to board
3. Open USB serial port, if you see “sd_status=3” the logger started
4. After a while (30 seconds or as long as you want) send ‘C’ for close to the serial monitor
5. sd_status=2 must show.
6. Put the sdcard in your computer. Open the csv file. Column N shows the time in took to write the previous line to sdcard. (in us).
7. Plot the write time, attached as image is the graph for mine. Every 256kb it takes >100ms to write.
Image
#47259
PSIG wrote:
Tue Dec 22, 2020 9:45 pm
I agree, there is less demand for SD logging, as other devices can be easily used for that purpose, such as phones. Still, it would be convenient to have a background full-time logging system that could catch and save unexpected events for analysis. Overwriting older log files as the card fills could allow this.
I agree, that is why is started this all :-D. triggered logging with some fault. Very fast logging! Always on logging. etc. Very nice to have.
PSIG wrote:
Tue Dec 22, 2020 9:45 pm
There have been multiple attempts at SD logging, without good success. @dazq had similar results years ago, solved by calling the data slowly so the SD could be written, and so a very slow capture rate. Let me ask a stupid question, to see if a solution is as simple as I hope. :roll:
:-D. If it where only that simple. Its difficult because all easy FAT file system access libs on arduino are blocking (to my knowledge). And it takes too much time to write to the sdcard at some point. (99% of the time it is ok) This is only solvable by modifying the libs to be polling/async style. An external logger would be easy however and no problem.
PSIG wrote:
Tue Dec 22, 2020 9:45 pm
It the problem that we are trying to write to SD directly, without a large buffer? TunerStudio and MegaLogViewer have no issues retrieving data smoothly for dashboard and logging at maximum rate, but they collect the data in a buffer (memory file), then write to a card (if desired) as a separate following step. Could it be that we need an SD module that retrieves data to a memory buffer (F-RAM chip?). That stored data is then written to SD as slowly or quickly as it wants, without interrupting Speeduino. I'm wondering if a multi-MB data buffer is one solution, or at least not trying to write directly to SD.Image
Its not the retrieval of data that is an issue. Its inside the speeduino firmware. Also buffer size is not an issue i tried to write in chuncks of 512 bytes to 4kbyte. All with the same result on the 256k boundary. The issue is that it needs to work inside the speeduino firmware. And no single call to a function may take more than ~1 or 2 milliseconds because you drop your loops/s to zero effectively. And no more updates on ignition and injection schedules, stopping speeduino from working. External logging is no issue. But that requires extra hardware.
#47261
LPG2CV wrote:
Wed Dec 23, 2020 8:43 am
Not a full solution. The quality of the SD card may be in question as well.
True that is indeed the case. But still at some point it would be more than the max ~3ms or so to write to the sdcard. That has everything todo with how these work internally. And non blocking/async/DMA access to the sdcard solves it. But that is not easy available as open source for arduino.
#47264
Let me restate my approach. Forget the SD card. Let Speeduino deposit data to an F-RAM chip in the proto area whenever it can without disturbing timing. Now the maximum-resolution data is permanently saved on memory chip. Done.

At this point or a month later, you can ignore it, or save it to another device, or overwrite it, etc. Have an SD module read it, or a phone, or whatever, at any speed, without disturbing Speeduino. Only read it if you want it, to any other device or storage. As the chip fills, it can overwrite older data. Is that making sense to remove the primary blocking and timing issues by saving data to chip, then worry about slow SD writing later?
#47268
You can do that. I would use a flash ic for that. Much larger sizes. and more than fast enough. No need for all the difficult stuff that is needed for EEPROM emulation because you just write it sequentially while logging. There is even 1 Mbyte left on the spi flash if it is used for EEPROM.

If you go that route even RAW logging on sdcard is possible. You just do not care about a file system on the card, bypassing all the issues i just described. I don't like the solution as you always need dedicated computer and software to read/translate the log to something useful. Would never be my personal preferred route.
#47272
Why not broadcast all the live data over canbus, then it can be displayed and even logged by an external device with much looser timing requirements. Plus, if it's over canbus you could install the SD card in an easier to access location too.

If you are going through the trouble of making the filesystem and individual files (yes!) then it makes sense to be able to swap the card. Otherwise, you may as well do as Psi recommends and just save to flash and download/convert at that time.

Or, both!

As for data, I think MS3 logs something like 512bytes at up to 100hz. That already has some limitations, and it'd be nice to avoid that. So, figure 25kb/s and 1Mb is looking pretty dang tiny. Personally, writing a larger buffer more often works best on a teensy for me. I think I am writing 2K chunks of all 512 MS3 canbus broadast channels at 100Hz, I forget but I think it's like 1.4K per broadcast? Seems to work pretty damn good, I do not use MS3's SD logger any more, because it's a buggy mess.
36-3 (ish) trigger wheel

Thanks again.

Many engines require the distributor to be retaine[…]

1.4 T-jet Centos

... It is a little bit arrogant to expect a Spee[…]

I am looking for information to adapt an iac step[…]

Still can't find what you're looking for?