Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.
|
Forum Index : Microcontroller and PC projects : Efficiently write a float array to .WAV file
Author | Message | ||||
leoec Newbie Joined: 13/11/2023 Location: United StatesPosts: 6 |
Hi All, I am new to PicoMite, Pi Pico, MMbasic, and this forum. I do have other experience with other microcontrollers and code environments. I'm a little over enthusiastic for a "workstation" that can run on like 50mA (excluding monitor) and welcome the associated limits -- mostly :). I've put together a PicoMiteVGA and got busy creating an ADC recording program to write ADC samples, taken at audio frequency and converted to a .WAV file on SD card. It has generally gone well after I wrote a hexdump.bas program to see a meaningful view of the file contents. I didn't see existing programs for either hexdumps nor recording .WAV files. Are there any examples out there? Now, I am running the code on a PicoMite(NO-VGA). When the sampling rate gets to around 8kHz the code runs into a bottleneck writing the samples to SD card before the next sample buffer is filled by the ADC. I assume this problem exists on the VGA version too. I am running the ADC in free running mode. I've tried writing the code a number of different ways in MMBasic without dipping my toes into CSUBs or other machine code. Below is the relevant part of the code I currently use that seems to function acceptably up to 8kHz with the PicoMite running at 133MHz. I'm looking to achieve a continuous sampling rate of at least 20kHz while writing all samples to SD. First, here are some questions I have: * The loop around bin2string() (below) can't be the most efficient option. Is there a more efficient construct in MMbasic to translate an integer buffer with elements between 0 and 255 to a byte string for writing to file? * Of course I was all over "MEMORY PACK sample_scaled,sample_scaled_packed,255,8" But ** the syntax never worked out and I tried it a lot of different ways. ** I saw conflicting usages of packed arrays in the PicoMite documentation and no references to it in the MicroMite/Learning to program/and MMbasic documents. I see that longstrings have a lot in common with packed arrays. Did longstrings replace packed arrays? ** If there is a way and I got it packed, how would I convert it to a string for printing to a file? * I tried a few different approaches using longstrings. All failed to yield a useful result. I see the "LONGSTRING APPEND..." and "LONGSTRING PRINT..." commands. Is there an efficient way to use some of the longstring commands to achieve efficient writing of bytes to a SD file? * What is the ideal number of bytes to write to SD in a single print statement to achieve the maximum throughput for a write? There were so many different ways to do this, that I'm thinking I got lost in the weeds and missed something obvious. Any suggestions for decreasing the time it takes to write an array of ADC samples to SD as a stream of bytes are much appreciated. Note, I haven't tried just writing the numbers in the array as text because my goal to record directly from ADC to a binary format .WAV continuously, without stopping, indefinitely. Eventually, I want a version of this code to run on a PicoMiteWEB and offer a livestream of a VLF receiver audio it is sampling. The relevant code snippet (There is a lot of code around this, the _sam_ready() interrupt handler subroutine and a couple of flip-flop buffers getting alternately read or written from.): [...] dim integer sambufsize = 255 ' Limited by string limit ADC open samrate, channum, _sam_ready() ' Setup the channel for capture Dim integer afcap_scaled(sambufsize) ' destination for MATH SCALE Dim string writebufstr ' 255 byte buffer to write samples in blocks Do If samready = 1 Then samready = 0 writebufstr = "" ' clear write buffer ' I write CAT; save, open, I see Inc instead. Works. For idx = 1 To sambufsize Inc writebufstr,Bin2str$(uint8,afcap_scaled(idx)) Next idx Print #1; writebufstr EndIf If repeatcount = samrepeat Then Exit ' Have all the samples been captured? Loop Print "recording complete." [...] |
||||
zeitfest Guru Joined: 31/07/2019 Location: AustraliaPosts: 482 |
It's a bit quiet !? From what I've gathered over the years, for speedy continuous A-D work it would be better to avoid the pico/MMbasic/averaging AD rigmarole and use an external A-D with, say, spi i/o. I guess buffers and interrupts would still be needed, not sure if the conversions and handling are fast enough overall though for your task. As well, the pico does not have any floating point accelerator, in my experience it bogs down a lot on FP, so it would be best to keep the data integer based. [ Disclaimer - I don't use MMbasic and don't have the projects it runs on, not a fan really ] |
||||
JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 3802 |
You could time different writes to see what's fastest but my guess would be a multiple of the SD card block size. You could also see whether longstring etc are faster. Bear in mind the ADC is poor (hardware bug) - but maybe good enough for you. You can overclock the Pico a lot :) John |
||||
matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 9122 |
First you must use the ADC command in background mode. ADC RUN buffers values 0-4095 to double buffers. Then use MATH SCALE to convert to 16 bit and BIN2STR to write out. I'll look at adding a block write capability if that doesn't work Edited 2023-11-18 18:19 by matherp |
||||
leoec Newbie Joined: 13/11/2023 Location: United StatesPosts: 6 |
zeitfest Thanks for breaking the silence :) Good to have that perspective that you don't know of continuous, high sampling rate A->D->(math transform)->(SD file) At the outset I recognized it was a stretch, but I saw that a lot of work has gone into the MMBasic API around the uC's features. I had hopes. Floating point: Indeed, I hoped the just have the A->D samples returned in an integer array from the API (which would be all machine code) but it uses only a float array according to the doc. Better yet, if ADC open had an argument to specify sample size in bits; either 12 bit or 8 bit and allow either a packed array or longstring (if packed arrays aren't supported any longer) for the output of raw integer samples instead of the floating point voltage. The option for raw numbers is good in my work -- less transformation, less error uncertainty, fewer resources consumed. I like your suggestion of an external ADC. That will likely land the sample results in a byte oriented format for storing directly. I know also that using an external ADC I would be able to specify bit depth and not need to scale 12bit samples to fit in an 8bit WAV. I have some MCP3008s which can easily sample 50kHz at 3.3v. I'll likely try those out at some point after I exhaust my options with the built in hardware. Thank you! |
||||
leoec Newbie Joined: 13/11/2023 Location: United StatesPosts: 6 |
JohnS That is exactly what I was wondering: what blocksize is transferred to the SD card. Is it 8 bits at a time, 16, 32? etc. One easy test you've made me think of is just try switching to an even number. Of course others as well. I'll put overclocking is on my list of possible remedies. I'm hoping to find a more efficient technique. However, that may be a really good test. Hardware issues with Pico ADC: I've read about that. There is an excellent series of characterization experiments documented by someone... Here it is: https://pico-adc.markomo.me/ In truth, I'm not sure the ADC in the pico will be sufficient. But, I wanted to explore the native MMBasic tools and PicoMite hardware as I really like understanding what the platform has before I get out my soldering iron :) Thanks for those ideas! |
||||
leoec Newbie Joined: 13/11/2023 Location: United StatesPosts: 6 |
matherp Considering what you, zeitfest, and JohnS have said, I think my biggest losses in efficiency right now are: 1) the need to convert an array of integers (64bits) to an array of bytes (currently only a single 255 byte string) using Bin2str() in a for loop. Please see below for the loop including Bin2str() and well as the full program further down. Bin2str() operates one byte at a time, if I understand correctly. 2) The requirement to capture the ADC samples to a float array instead of a correctly sized array of UINT8 (for 8 bit ADC samples) or UINT16 (for 12 bit ADC samples). I know currently the only option is 12 by samples. More detail on your specific points I think I am using the ADC that way now; I called it "free running mode" but I didn't provide that part of the code. My bad! If I'm not using it right, please advise :) Here is the ADC init I'm using (full code with interrupt handler below): ADC open samrate, channum, _sam_ready() ' Setup the channel for capture ADC start afcap_0() ' Start capturing into bufnum 0 My understanding of the documentation is that ADC open requires a float array and ADC start <float_array> will fill that array with samples captured at the sampling rate. I am currently using MATH SCALE to transform from a float array (values in the range 0-3.3) to an int array (balues in the range 0 - 255), like this: Math scale afcap_0(),77,afcap_scaled() ' 77 =~ 255/3.3 as integer I believe I loose the most speed in this loop converting the 64bit int array of scaled samples to a byte array for writing to SD. That looks like this: For idx = 1 To sambufsize Inc writebufstr,Bin2str$(uint8,afcap_scaled(idx)) ' Exiting edit converts CAT to INC Next idx Print #1; writebufstr For reference, the full (buggy and likely armed and dangerous) program, which does appear to mostly work except for the dropping of samples, is: ' adc2wav v19 drops samples over about 6kHz samrate ' by Leo Edmiston-Cyr <leoec@gopher.quest> ' Last updated: 11/11/2023 ' Option explicit ' require explicit variable declaration Dim samdelay = 0 ' duration to delay before recording Dim samduration = 30 ' duration of recording in seconds Dim sambufsize = 255 ' string limit -- buffer size for recrd/writing to file Dim samrate = 10000 ' samples per second (Hz) Dim samnum = samduration * samrate ' total samples in recording Dim integer samrepeat = Fix(samnum /sambufsize) ' number sambufsize ADC cycles Dim string filename = mm.cmdline$ If filename = "" Then filename = "adc2wav-" + Date$ + Str$(Epoch(now)) + ".wav" EndIf ' ' RIFF WAVE format metadata ' samnum <-from above Dim samsize = 1 ' bytes per sample Dim channum = 1 ' 1 = Mono; 2 = Stereo ' samrate <-from above 'compute chunk size 'Dim integer chunksize = 32 + samnum * samsize * samrepeat ' num samples * Dim integer chunksize = 32 + samnum * samsize ' num samples * Dim integer subChunkSize = 16 Dim integer byterate = samrate * channum * samsize Dim integer sambits = samsize*8 '8 bits per byte; obs Dim integer idx ' When you need a loop index counter... Just in case ;) ' Hi user, here's what's going on Print "Recording to:",filename Print "duration:",samduration Print "channel(s):",channum Print "bit depth:",samsize*8;"bits" Print "sample rate:",samrate;"Hz" Print "buffer size:",sambufsize;"byte(s)" Print "buffer writes:",samrepeat Print "total samples:",samnum ' ' write WAVE file format for later .wav playback Open filename For output As #1 'The following metadata satisfies the RIFF WAVE format requirements Print #1; "RIFF"; Print #1; Bin2str$(UINT32,chunksize); Print #1; "WAVE"; Print #1; "fmt "; ' SubChunk Print #1; Bin2str$(UINT32,subChunkSize); ' the size of this sub chunkfor PCM Print #1; Bin2str$(UINT16, 1); ' Audio format PCM = 1 is linear quantization Print #1; Bin2str$(UINT16, channum); ' number of channels 2 = stereo Print #1; Bin2str$(UINT32, samrate); ' sample rate 8000, 44100, etc. Print #1; Bin2str$(UINT32, byterate); Print #1; Bin2str$(UINT16, channum * samsize); ' block align-bytes for one sample of all channels Print #1; Bin2str$(UINT16, sambits); ' bits per sample ' ExtraParams -- none for PCM encoding ' ExtraParamSize -- doesn't exist for PCM ' ExtraParams Print #1; "data"; ' Subchunk2ID ' RIFF WAVE metadata is complete Dim float afcap_0(sambufsize) ' required destination for ADC start Dim float afcap_1(sambufsize) ' used with _a in a sampling-writing sequence Dim integer afcap_scaled(sambufsize) ' destination for MATH SCALE Dim integer samp Dim integer samrunning = 0 Dim integer repeatcount = 0 Dim integer samready = 0 ' Flag for ADC completion when the sample is ready Dim integer samwritten = 1 ' flag if writing has completed Dim integer intsmissed = 0 ' count of missed interrupts ' Initiate sampling of the audio input SetPin gp26, ain ' set ADC input ADC open samrate, channum, _sam_ready() ' Setup the channel for capture ADC start afcap_0() ' Start capturing into bufnum 0 Dim bufnum = 0 ' which buffer is being sampled into? Dim string writebufstr ' 255 byte buffer to write samples in blocks Do If samready = 1 Then samready = 0 writebufstr = "" ' clear write buffer 'For idx = 1 To sambufsize ' Print #1; Bin2str$(uint8,afcap_scaled(idx)); 'Next idx ' I write CAT, save, open, I see Inc instead For idx = 1 To sambufsize Inc writebufstr,Bin2str$(uint8,afcap_scaled(idx)) Next idx Print #1; writebufstr EndIf If repeatcount = samrepeat Then Exit ' Have all the samples been captured? Loop Print "recording complete." Close #1 ADC close ''''''''''''''''''''''''''''''''''''''' 'sub pattern reference 'sub mysub arg1, arg2$, arg3 'end sub 'NOTE: No parameters can be passed to an interrupt handler subroutine ''''''''''''''''''''''''''''''''''''''' '''''''''''''''''''''''''''''''''''''''' ' Interrupt handling with flip-flop buffers for "ADC open <subname>" ''''''''''''''''''''''''''''''''''''''' Sub _sam_ready() If repeatcount = samrepeat Then End ' End if all samples are captured ' this is the only place where knowledge of the active buffer is needed If bufnum = 0 Then ' If bufnum 0 is full, start next capture into bufnum 1; scale for write ADC start afcap_1() Math scale afcap_0(),77,afcap_scaled() ' 77 =~ 255/3.3 as integer bufnum = 1 Else ' bufnum 1 is full, start capture into 0 and scale for write ADC start afcap_0() Math scale afcap_1(),77,afcap_scaled() ' 77 =~ 255/3.3 as integer bufnum = 0 EndIf samrunning = 1 Inc repeatcount samready = 1 samrunning = 0 End Sub |
||||
matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 9122 |
Try this, seems to work perfectly for me Option explicit Option default integer Const buffsize = 1000 '4000 samples Const nsamples = buffsize * 4 Dim a(buffsize-1),b(buffsize-1),d(nsamples-1) ' print the wav header to the file here ADC open 20000,1,myint 'one channel, sampled at 20000Hz" Dim i,j,k,c=0, count=0 Dim s$ Open "B:/data.wav" For output As #1 ADC run a(),b() Do Loop Until c 'wait for the first conversion to complete Do If c=1 Then Memory unpack a(),d(), nsamples, 16 Else Memory unpack a(),d(), nsamples, 16 EndIf Math scale d(),16,d() 'was 0-4095 scale to 0 to 65520 i=0 Do While i<buffsize s$="" j=Min(50,nsamples-i) For k=0 To j-1 Inc s$,Bin2str$(uint16,d(i+k)) Next Inc i,j Print #1,s$ If j<>50 Then Print j Loop Inc count Print c; Timer =0 c=0 Do :Loop Until c Print Timer 'see how much time we have left before we need to process again Loop Until count=100 ' arbitrary duration Close #1 ' Sub myint c= MM.Info(ADC) End Sub OPTION SYSTEM SPI GP10,GP11,GP12 OPTION COLOURCODE ON OPTION CPUSPEED 252000 'KHz OPTION SDCARD GP15 OPTION AUDIO VS1053 GP2,GP3,GP4,GP5,GP6,GP7,GP8', ON PWM CHANNEL 1 |
||||
Bleep Guru Joined: 09/01/2022 Location: United KingdomPosts: 509 |
I believe that second Memory unpack should be 'b()' I think? If c=1 Then Memory unpack a(),d(), nsamples, 16 Else Memory unpack b(),d(), nsamples, 16 EndIf Edited 2023-11-19 08:11 by Bleep |
||||
leoec Newbie Joined: 13/11/2023 Location: United StatesPosts: 6 |
bleep: Agreed. I noticed that. matherp: That combination of: ADC run and the packed arrays should have a huge impact on performance. I spent a little time scratching my head on syntax and other errors trying to run your code on 0707 and then decided to check 0708rc19's changelog and I see some of those features referenced. I tried to upgrade and retest but I'm having some issue getting my sd card setup again in 0708. Those new features look amazing for this kind of work. I'll figure that out and let you know how it goes with the changes under 0708rc19. Thank you. |
||||
matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 9122 |
Please could you try this new version (RC20) and the new Basic approach and let me know results - thanks PicoMite.zip Option explicit Option default integer Const buffsize = 1000 '4000 samples Const nsamples = buffsize * 4 const nbytes = buffsize * 8 Dim a(buffsize-1),b(buffsize-1) ADC open 20000,1,myint 'one channel, sampled at 20000Hz" Dim c=0, count=0 Dim s$ Open "B:/data.wav" For output As #1 ' print the wav header to the file here ADC run a(),b() Do Loop Until c 'wait for the first conversion to complete Do If c=1 Then Math shift a(),4,a() ' shift all array elements 4 bits left = multiply by 16 Memory print #1,nbytes,a() 'print array direct to disk Else Math shift b(),4,b() Memory print #1,nbytes,b() EndIf Print c; Timer =0 c=0 Do :Loop Until c Print Timer 'see how much time we have left before we need to process again Inc count Loop Until count=100 ' arbitrary duration Close #1 ' Sub myint c= MM.Info(ADC) End Sub Edited 2023-11-19 23:50 by matherp |
||||
leoec Newbie Joined: 13/11/2023 Location: United StatesPosts: 6 |
Your new code runs straight away under rc20 without modification. However, to play it I need the wave header or other visualization tools I haven't crafted. I completed a thoughtful integration of your ADC routines into my existing adc2wav code and I get mostly noise. It does bear some similarity in periodicity in common with the sampled radio signal. I don't believe it is anything as simple as volume or an a()<->b() buffer reversal. I didn't have enough time to dive deep enough to figure out what is going wrong. I am preparing for and traveling through next weekend so this is about as far as I'll get until I have time after getting back. My gut is that there is something wrong with the ADC run <array1>,<array2> output into those arrays. Like they are aligned on an odd byte boundary or something. Or maybe stored in big endian? (IIRC WAVE is expecting little endian for the data). Oh, also of note, I did comment out the x16 scaling shift (and play with different shifts) and sometimes had something closer to the audio source. Finally, my original wave writing code worked for sure to write 8bit samples in WAV format. I believe I modified it correctly to save 16 bit samples. However, that is another undertainty. When I get back my next step is probably to choose some very clean and predictable waveforms, like a 220Hz sine or square wave and look at the hexdump. Not to mention bring a fresh perspective to it. Just in case you or anyone else wants to experiment with the WAV output, my updated adc2wav.bas code is below: ' adc2wav v23 includes pmather ADC routines ' by Leo Edmiston-Cyr <leoec@gopher.quest> ' Last updated: 11/20/2023 ' Option explicit Option default integer Dim c=0, count=0 ' print the wav header to the file here Dim samdelay = 0 ' duration to delay before recording Dim samduration = 10 ' duration of recording in seconds Const sambufsize = 1000 ' 4000 16 bit samples; 12 bit resolution Const sams_in_buf = sambufsize * 4 ' 4 16 bit samples per int Const nbytes = sambufsize * 8 ' 8 bytes per int Dim samrate = 20000 ' samples per second (Hz) Const samnum = samduration * samrate ' total samples in recording Const samrepeat = samnum / sams_in_buf ' num sambufsize ADC cycles Dim string filename = mm.cmdline$ If filename = "" Then filename = "adc2wav-" + Date$ + Str$(Epoch(now)) + ".wav" EndIf Dim samsize = 2 ' bytes per sample Dim channum = 1 ' 1 = Mono; 2 = Stereo Dim chunksize = 32 + samnum * samsize ' num samples * Dim subChunkSize = 16 Dim byterate = samrate * channum * samsize Dim sambits = samsize * 8 '8 bits per byte; obs Dim afcap_0(sambufsize-1), afcap_1(sambufsize-1) ' Hi user, here's what's going on Print "Recording to:",filename Print "duration:",samduration Print "channel(s):",channum Print "bit depth:",samsize*8;"bits" Print "sample rate:",samrate;"Hz" Print "buffer size:",sambufsize;"byte(s)" Print "buffer writes:",samrepeat Print "total samples:",samnum ' ' write WAVE file format for later .wav playback Open filename For output As #1 'The following metadata satisfies the RIFF WAVE format requirements Print #1; "RIFF"; Print #1; Bin2str$(UINT32,chunksize); Print #1; "WAVE"; Print #1; "fmt "; ' SubChunk Print #1; Bin2str$(UINT32,subChunkSize); ' the size of this sub chunkfor PCM Print #1; Bin2str$(UINT16, 1); ' Audio format PCM = 1 is linear quantization Print #1; Bin2str$(UINT16, channum); ' number of channels 2 = stereo Print #1; Bin2str$(UINT32, samrate); ' sample rate 8000, 44100, etc. Print #1; Bin2str$(UINT32, byterate); Print #1; Bin2str$(UINT16, channum * samsize); ' block align-bytes for one sample of all channels Print #1; Bin2str$(UINT16, sambits); ' bits per sample ' ExtraParams -- none for PCM encoding ' ExtraParamSize -- doesn't exist for PCM ' ExtraParams Print #1; "data"; ' Subchunk2ID ' RIFF WAVE metadata is complete ' Initiate sampling of the audio input ''''''''''''''''''''''''''''''''''''''' ADC open samrate,channum,myint ADC run afcap_0(),afcap_1() Do Loop Until c 'wait for the first conversion to complete Do If c=1 Then Math shift afcap_0(),4,afcap_0() ' shift all array elements 4 bits left = multiply by 16 Memory print #1,nbytes,afcap_0() 'print array direct to disk Else Math shift afcap_1(),4,afcap_1() Memory print #1,nbytes,afcap_1() EndIf 'Print c; Timer =0 c=0 Do :Loop Until c 'Print Timer 'time remaining before next sample is ready to write Inc count Loop Until count=samrepeat ' arbitrary duration Close #1 Print "recording complete." ADC close Sub myint c= MM.Info(ADC) End Sub |
||||
matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 9122 |
Best thing is to decompose this until you find my or your error. Start with just the ADC RUN command and then unpack the array and plot it manually or using the new LINE PLOT command Assuming that looks OK do the same again after first using the shift before the unpack. Once that is all OK then you can look at the WAV header. NB: I think you are missing characters 41-44 in the header which certainly aren't optional when sending data to the VS1053 |
||||
Print this page |