[Tjeerd]

Why not broadcast all the live data over canbus, then it can be displayed and even logged by an external device with much looser timing requirements. Plus, if it's over canbus you could install the SD card in an easier to access location too.

As mentioned before, all true. But needs extra hardware and that was not the point. The extra hardware does not mind that once every 10 or 100 seconds it runs for 100ms vs 2ms. That is only problematic inside the speeduino firmware. The hardware for sdcard is already on the board. So hoped to use that instead of using a separate hardware design + separate TS interface to configure it? Or some other interface?

If you are going through the trouble of making the filesystem and individual files (yes!) then it makes sense to be able to swap the card. Otherwise, you may as well do as Psi recommends and just save to flash and download/convert at that time.

Or, both!

Well getting TS to get the data transmitted to the computer with the ms3 interface is not documented (as far as i know). So its not really easy to implement. If you have some docs please share! Making the file system approach + swap able sd seemed easy enough, until the 2 problems. I am still contemplating converting the system to DMA or interrupt based. But lots of work. And not enough time to work on it.

As for data, I think MS3 logs something like 512bytes at up to 100hz. That already has some limitations, and it'd be nice to avoid that. So, figure 25kb/s and 1Mb is looking pretty dang tiny. Personally, writing a larger buffer more often works best on a teensy for me. I think I am writing 2K chunks of all 512 MS3 canbus broadast channels at 100Hz, I forget but I think it's like 1.4K per broadcast? Seems to work pretty damn good, I do not use MS3's SD logger any more, because it's a buggy mess.

It could run 2 Mbyte/s average easy even without DMA. With DMA and biger chuncks to the sdcard it should do more depending on the speed of the sdcard. The issue is not the data rate, it really only is the inconsistent long delay that sometimes happens to the write command. That delay is not allowed by the current speeduino firmware design (stopping the main loop for that amount of time). I tried a Buffer as big as 30K, writing chunks of 4K. Does not matter it all works, but the blocking delay of a single write takes >100ms at some point. Only changing the system to be able to stop/pauze the transfer and continue later works to use it inside the speeduino firmware. Separate threads would be really nice here . But for that we need to run an RTOS etc. All also complicated stuff .

If you have a teensy with sdcard logger can you please log the time all writes take? I am curious if the teensy already does something clever or that it has the same sporadic long write times.

[moonie223]

I've got a log I pulled from a 2.8 cummins using a teensy right here and ready, it uses the same logger as I'm using on ms3.



Looks like I get some long writes in there too, over 250ms, but I think the large majority of the buffer writes only take about 20-25ms.

I am pretty sure the 250ms delay is my logger getting confused on j1939 extended frames. It only hit 6 times in ~50k canbus transmissions.

You are using SDIO right, no spi?

[hindsight]

This thread seems to be coincidental - I recently came to the conclusion that a separate logging device would be useful, and I've started to write the software for a Teensy 3.5 to connect to the secondary Serial of my MEGA2560, retrieve the data, then to write the data into MSVLG format on the onboard SD card, ready for megalogviewer.

Very early stages at the moment - literally just started it a few days ago, prior to seeing this message, but happy to share if there is interest.

It's less than ideal obviously to have two microcontrollers, but it does fully remove any potential impact of doing the logging on the main controller, it could also lend itself to having a GPS device on the secondary (logging) controller, which I'm guessing could be used to keep a RTC accurate.

[Tjeerd]

I've got a log I pulled from a 2.8 cummins using a teensy right here and ready, it uses the same logger as I'm using on ms3.

Looks like I get some long writes in there too, over 250ms, but I think the large majority of the buffer writes only take about 20-25ms.

I am pretty sure the 250ms delay is my logger getting confused on j1939 extended frames. It only hit 6 times in ~50k canbus transmissions.

So far i have tested a couple of sd cards, and all pull the SDIO D0 low for 100 to 200ms to indicate the sd controller is busy (sd card not ready). The STM32 sdcard implementation did only return after this internal stuff is done after a write. I also suspect most other implementations do something similar. This would be the preferred way of doing stuff usually.

I now changed that to return immediately and check if sd card is ready BEFORE a write/ready/update. This works very nice! Now i log in ram buffer when sdcard is busy and keep speeduino running until sd card is free again and write after that. I also enabled HW flow control to prevent FIFO under-run problems i did have.

A all night test worked nice with a resulting 220mbyte file with no hick ups using speeduino firmware.

You are using SDIO right, no spi?

yes i do. 4 bit wide buss 48mhz clock.

[Tjeerd]

This thread seems to be coincidental - I recently came to the conclusion that a separate logging device would be useful, and I've started to write the software for a Teensy 3.5 to connect to the secondary Serial of my MEGA2560, retrieve the data, then to write the data into MSVLG format on the onboard SD card, ready for megalogviewer.

I don't know about you but for me it is corona holiday time. So i had something todo .

Very early stages at the moment - literally just started it a few days ago, prior to seeing this message, but happy to share if there is interest.

If you have something that works with MSVLG format i would like to try and put that into the speeduino too. I couldn't find any info with a quick google search so i used CSV for now. MLV can handle that too. When that is finished i can always search for that info.

[ric355]

For those wanting an external logging device that runs permanently, a reminder that speedylogger exists and produces files suitable for loading directly into MegaLogViewer. It takes data via serial3 at about 25hz. As soon as power is applied it starts logging and can be left as a permanent logging solution. Works fine on a pi zero.

https://github.com/ric355/SpeedyLogger

[hindsight]

If you have something that works with MSVLG format i would like to try and put that into the speeduino too. I couldn't find any info with a quick google search so i used CSV for now. MLV can handle that too. When that is finished i can always search for that info.

Yeah - I have something now that works directly with MSVLG format. Not everything is implemented yet, but I have a teensy 3.5 connected to the secondary serial and generating MSVLG format files that can be natively read by megalogviewer. The file format is described here..

http://www.efianalytics.com/TunerStudio ... at_1.0.pdf

I'll pop what I have up onto GitHub later, and you can have a look. It's still pretty rough at the moment though (will confirm when it's available).

For those wanting an external logging device that runs permanently, a reminder that speedylogger exists and produces files suitable for loading directly into MegaLogViewer. It takes data via serial3 at about 25hz. As soon as power is applied it starts logging and can be left as a permanent logging solution. Works fine on a pi zero.

https://github.com/ric355/SpeedyLogger

Interesting, thanks - I have a zero W sitting here also, so will take a look. It would be particularly neat if the zeroW could also act as a WiFi access point, and the log files shared via SMB or HTTP - no mucking about with SD cards. I see that you've used Ultibo for this, not clear if this would be possible with it - do you know?

Mike

[ric355]

Interesting, thanks - I have a zero W sitting here also, so will take a look. It would be particularly neat if the zeroW could also act as a WiFi access point, and the log files shared via SMB or HTTP - no mucking about with SD cards. I see that you've used Ultibo for this, not clear if this would be possible with it - do you know?

Mike

Unfortunately only wired networking works on Ultibo at the moment. WiFI will come eventually but there is no schedule for it at present.

My car dash uses several Pi's running Ultibo apps (a 2, a 3b+ and a ZeroW) and I do use these with WiFI for updating software and accessing logs but I have done it by designing a means to dual boot between an Ultibo and a Linux kernel. I use a canbus command which originates from a press on a touch screen to initiate the swap. When I'm done I swap back to Ultibo with a simple shell script. This mode change between Ultibo and PiOS is just replacing the config.txt file with one that points to a different kernel, so is very quick and non-destructive.

If you had only a single zeroW and non canbus, my touch screen command could be replaced by adding a momentary button to a GPIO to initiate the same process.

That button thing reminds me there's another thing speedylogger can do: If you add a momentary push button to a GPIO pin (I forget which one but it should be obvious in the code) then whenever you press it SpeedyLogger will add a log file marker to the log file. These will show up in MLV as vertical red lines on the charts. This is one of the things I wanted when I wrote it - permanent logging and the ability to mark a spot in the log if something odd happened while driving.

[hindsight]

That button thing reminds me there's another thing speedylogger can do: If you add a momentary push button to a GPIO pin (I forget which one but it should be obvious in the code) then whenever you press it SpeedyLogger will add a log file marker to the log file.

I spotted reference to those markers in the documentation, definitely sounds like a useful facility.

I've just published my code at https://github.com/minceheid/Speeduino-Copilot. It's still very rough, as I really only started looking at this a few days back, but it's largely functional.

[Tjeerd]

Yeah - I have something now that works directly with MSVLG format. Not everything is implemented yet, but I have a teensy 3.5 connected to the secondary serial and generating MSVLG format files that can be natively read by megalogviewer. The file format is described here..

http://www.efianalytics.com/TunerStudio ... at_1.0.pdf

I'll pop what I have up onto GitHub later, and you can have a look. It's still pretty rough at the moment though (will confirm when it's available).

Thanks! i most definitely did not search good enough . I will go on with this as CSV logging was at least working. I will make the internal sdcard log triggered too, (external pin, alarms, coolent temp based etc). Also saving stuff in a ring buffer until the trigger so the events leading up to it are also saved and logged. But all work in progress and not progressing very quickly at the moment.
[/quote]