Home
JAQForum Ver 24.01
Log In or Join  
Active Topics
Local Time 07:35 27 Nov 2024 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Efficiently write a float array to .WAV file

Author Message
leoec
Newbie

Joined: 13/11/2023
Location: United States
Posts: 6
Posted: 02:42am 15 Nov 2023
Copy link to clipboard 
Print this post

Hi All,

I am new to PicoMite, Pi Pico, MMbasic, and this forum.  I do have other experience with other microcontrollers and code environments.  I'm a little over enthusiastic for a "workstation" that can run on like 50mA (excluding monitor) and welcome the associated limits -- mostly :).

I've put together a PicoMiteVGA and got busy creating an ADC recording program to write ADC samples, taken at audio frequency and converted to a .WAV file on SD card.  It has generally gone well after I wrote a hexdump.bas program to see a meaningful view of the file contents.  I didn't see existing programs for either hexdumps nor recording .WAV files.  Are there any examples out there?

Now, I am running the code on a PicoMite(NO-VGA).  When the sampling rate gets to around 8kHz the code runs into a bottleneck writing the samples to SD card before the next sample buffer is filled by the ADC.  I assume this problem exists on the VGA version too.  I am running the ADC in free running mode.  I've tried writing the code a number of different ways in MMBasic without dipping my toes into CSUBs or other machine code.

Below is the relevant part of the code I currently use that seems to function acceptably up to 8kHz with the PicoMite running at 133MHz.  I'm looking to achieve a continuous sampling rate of at least 20kHz while writing all samples to SD.

First, here are some questions I have:
* The loop around bin2string() (below) can't be the most efficient option.  Is there a more efficient construct in MMbasic to translate an integer buffer with elements between 0 and 255 to a byte string for writing to file?
* Of course I was all over "MEMORY PACK sample_scaled,sample_scaled_packed,255,8"  But
** the syntax never worked out and I tried it a lot of different ways.
** I saw conflicting usages of packed arrays in the PicoMite documentation and no references to it in the MicroMite/Learning to program/and MMbasic documents.  I see that longstrings have a lot in common with packed arrays.  Did longstrings replace packed arrays?
** If there is a way and I got it packed, how would I convert it to a string for printing to a file?
* I tried a few different approaches using longstrings.  All failed to yield a useful result.   I see the "LONGSTRING APPEND..." and "LONGSTRING PRINT..." commands.  Is there an efficient way to use some of the longstring commands to achieve efficient writing of bytes to a SD file?
* What is the ideal number of bytes to write to SD in a single print statement to achieve the maximum throughput for a write?

There were so many different ways to do this, that I'm thinking I got lost in the weeds and missed something obvious.  Any suggestions for decreasing the time it takes to write an array of ADC samples to SD as a stream of bytes are much appreciated.

Note, I haven't tried just writing the numbers in the array as text because my goal to record directly from ADC to a binary format .WAV continuously, without stopping, indefinitely.  Eventually, I want a version of this code to run on a PicoMiteWEB and offer a livestream of a VLF receiver audio it is sampling.

The relevant code snippet (There is a lot of code around this, the _sam_ready() interrupt handler subroutine and a couple of flip-flop buffers getting alternately read or written from.):

[...]
dim integer sambufsize = 255 ' Limited by string limit
ADC open samrate, channum, _sam_ready() ' Setup the channel for capture
Dim integer afcap_scaled(sambufsize) ' destination for MATH SCALE
Dim string writebufstr ' 255 byte buffer to write samples in blocks

Do
 If samready = 1 Then
   samready = 0
   writebufstr = ""  ' clear write buffer
   ' I write CAT; save, open, I see Inc instead.  Works.
   For idx = 1 To sambufsize
     Inc writebufstr,Bin2str$(uint8,afcap_scaled(idx))
   Next idx
   Print #1; writebufstr
 EndIf

 If repeatcount = samrepeat Then Exit  ' Have all the samples been captured?
Loop

Print "recording complete."
[...]
 
zeitfest
Guru

Joined: 31/07/2019
Location: Australia
Posts: 482
Posted: 01:17am 18 Nov 2023
Copy link to clipboard 
Print this post

It's a bit quiet !?

From what I've gathered over the years, for speedy continuous A-D work it would be better to avoid the pico/MMbasic/averaging AD rigmarole and use an external A-D with, say, spi i/o. I guess buffers and interrupts would still be needed, not sure if the conversions and handling are fast enough overall though for your task. As well, the pico does not have any floating point accelerator, in my experience it bogs down a lot on FP, so it would be best to keep the data integer based.

[    Disclaimer - I don't use MMbasic and don't have the projects it runs on, not a fan really     ]
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3802
Posted: 07:57am 18 Nov 2023
Copy link to clipboard 
Print this post

You could time different writes to see what's fastest but my guess would be a multiple of the SD card block size.

You could also see whether longstring etc are faster.

Bear in mind the ADC is poor (hardware bug) - but maybe good enough for you.

You can overclock the Pico a lot :)

John
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 9122
Posted: 08:18am 18 Nov 2023
Copy link to clipboard 
Print this post

First you must use the ADC command in background mode. ADC RUN buffers values 0-4095 to double buffers. Then use MATH SCALE to convert to 16 bit and BIN2STR to write out. I'll look at adding a block write capability if that doesn't work
Edited 2023-11-18 18:19 by matherp
 
leoec
Newbie

Joined: 13/11/2023
Location: United States
Posts: 6
Posted: 08:00pm 18 Nov 2023
Copy link to clipboard 
Print this post

zeitfest Thanks for breaking the silence :)  Good to have that perspective that you don't know of continuous, high sampling rate A->D->(math transform)->(SD file)  At the outset I recognized it was a stretch, but I saw that a lot of work has gone into the MMBasic API around the uC's features.  I had hopes.  Floating point: Indeed, I hoped the just have the A->D samples returned in an integer array from the API (which would be all machine code) but it uses only a float array according to the doc.  Better yet, if ADC open had an argument to specify sample size in bits; either 12 bit or 8 bit and allow either a packed array or longstring (if packed arrays aren't supported any longer) for the output of raw integer samples instead of the floating point voltage.  The option for raw numbers is good in my work -- less transformation, less error uncertainty, fewer resources consumed.

I like your suggestion of an external ADC.  That will likely land the sample results in a byte oriented format for storing directly.  I know also that using an external ADC I would be able to specify bit depth and not need to scale 12bit samples to fit in an 8bit WAV.  I have some MCP3008s which can easily sample 50kHz at 3.3v.  I'll likely try those out at some point after I exhaust my options with the built in hardware.

Thank you!
 
leoec
Newbie

Joined: 13/11/2023
Location: United States
Posts: 6
Posted: 08:10pm 18 Nov 2023
Copy link to clipboard 
Print this post

JohnS That is exactly what I was wondering: what blocksize is transferred to the SD card.  Is it 8 bits at a time, 16, 32? etc.  One easy test you've made me think of is just try switching to an even number.  Of course others as well.

I'll put overclocking is on my list of possible remedies.  I'm hoping to find a more efficient technique.  However, that may be a really good test.

Hardware issues with Pico ADC: I've read about that.  There is an excellent series of characterization experiments documented by someone... Here it is: https://pico-adc.markomo.me/

In truth, I'm not sure the ADC in the pico will be sufficient.  But, I wanted to explore the native MMBasic tools and PicoMite hardware as I really like understanding what the platform has before I get out my soldering iron :)

Thanks for those ideas!
 
leoec
Newbie

Joined: 13/11/2023
Location: United States
Posts: 6
Posted: 08:51pm 18 Nov 2023
Copy link to clipboard 
Print this post

matherp Considering what you, zeitfest, and JohnS have said, I think my biggest losses in efficiency right now are:

1) the need to convert an array of integers (64bits) to an array of bytes (currently only a single 255 byte string) using Bin2str() in a for loop.  Please see below for the loop including Bin2str() and well as the full program further down.  Bin2str() operates one byte at a time, if I understand correctly.

2) The requirement to capture the ADC samples to a float array instead of a correctly sized array of UINT8 (for 8 bit ADC samples) or UINT16 (for 12 bit ADC samples).  I know currently the only option is 12 by samples.

More detail on your specific points
I think I am using the ADC that way now; I called it "free running mode" but I didn't provide that part of the code.  My bad!  If I'm not using it right, please advise :)  Here is the ADC init I'm using (full code with interrupt handler below):

ADC open samrate, channum, _sam_ready() ' Setup the channel for capture
ADC start afcap_0() ' Start capturing into bufnum 0


My understanding of the documentation is that ADC open requires a float array and ADC start <float_array> will fill that array with samples captured at the sampling rate.  I am currently using MATH SCALE to transform from a float array (values in the range 0-3.3) to an int array (balues in the range 0 - 255), like this:

Math scale afcap_0(),77,afcap_scaled()  ' 77 =~ 255/3.3 as integer


I believe I loose the most speed in this loop converting the 64bit int array of scaled samples to a byte array for writing to SD.  That looks like this:

For idx = 1 To sambufsize
 Inc writebufstr,Bin2str$(uint8,afcap_scaled(idx)) ' Exiting edit converts CAT to INC
Next idx
Print #1; writebufstr


For reference, the full (buggy and likely armed and dangerous) program, which does appear to mostly work except for the dropping of samples, is:

' adc2wav v19 drops samples over about 6kHz samrate
' by Leo Edmiston-Cyr <leoec@gopher.quest>
' Last updated: 11/11/2023
'
Option explicit ' require explicit variable declaration

Dim samdelay = 0        ' duration to delay before recording
Dim samduration = 30    ' duration of recording in seconds
Dim sambufsize = 255   ' string limit -- buffer size for recrd/writing to file
Dim samrate = 10000      ' samples per second (Hz)
Dim samnum = samduration * samrate  ' total samples in recording
Dim integer samrepeat = Fix(samnum /sambufsize)  ' number sambufsize ADC cycles

Dim string filename = mm.cmdline$
If filename = "" Then
 filename = "adc2wav-" + Date$ + Str$(Epoch(now)) + ".wav"
EndIf

'
' RIFF WAVE format metadata
' samnum <-from above
Dim samsize = 1 ' bytes per sample
Dim channum = 1 ' 1 = Mono; 2 = Stereo
' samrate <-from above
'compute chunk size
'Dim integer chunksize = 32 + samnum * samsize * samrepeat ' num samples *
Dim integer chunksize = 32 + samnum * samsize ' num samples *
Dim integer subChunkSize = 16
Dim integer byterate = samrate * channum * samsize
Dim integer sambits = samsize*8 '8 bits per byte; obs

Dim integer idx ' When you need a loop index counter...  Just in case ;)

' Hi user, here's what's going on
Print "Recording to:",filename
Print "duration:",samduration
Print "channel(s):",channum
Print "bit depth:",samsize*8;"bits"
Print "sample rate:",samrate;"Hz"
Print "buffer size:",sambufsize;"byte(s)"
Print "buffer writes:",samrepeat
Print "total samples:",samnum

'
' write WAVE file format for later .wav playback
Open filename For output As #1

'The following metadata satisfies the RIFF WAVE format requirements
Print #1; "RIFF";
Print #1; Bin2str$(UINT32,chunksize);
Print #1; "WAVE";
Print #1; "fmt ";

' SubChunk
Print #1; Bin2str$(UINT32,subChunkSize);  ' the size of this sub chunkfor PCM
Print #1; Bin2str$(UINT16, 1); ' Audio format PCM = 1 is linear quantization
Print #1; Bin2str$(UINT16, channum); ' number of channels 2 = stereo
Print #1; Bin2str$(UINT32, samrate);  ' sample rate 8000, 44100, etc.
Print #1; Bin2str$(UINT32, byterate);
Print #1; Bin2str$(UINT16, channum * samsize); ' block align-bytes for one sample of all channels
Print #1; Bin2str$(UINT16, sambits);  ' bits per sample

' ExtraParams -- none for PCM encoding
'   ExtraParamSize -- doesn't exist for PCM
'   ExtraParams

Print #1; "data";  ' Subchunk2ID
' RIFF WAVE metadata is complete

Dim float afcap_0(sambufsize) ' required destination for ADC start
Dim float afcap_1(sambufsize) ' used with _a in a sampling-writing sequence
Dim integer afcap_scaled(sambufsize) ' destination for MATH SCALE

Dim integer samp
Dim integer samrunning = 0
Dim integer repeatcount = 0
Dim integer samready = 0        ' Flag for ADC completion when the sample is ready
Dim integer samwritten = 1      ' flag if writing has completed
Dim integer intsmissed = 0      ' count of missed interrupts


' Initiate sampling of the audio input
SetPin gp26, ain  ' set ADC input
ADC open samrate, channum, _sam_ready() ' Setup the channel for capture
ADC start afcap_0() ' Start capturing into bufnum 0
Dim bufnum = 0  ' which buffer is being sampled into?
Dim string writebufstr ' 255 byte buffer to write samples in blocks

Do
 If samready = 1 Then
   samready = 0
   writebufstr = ""  ' clear write buffer
   'For idx = 1 To sambufsize
   '  Print #1; Bin2str$(uint8,afcap_scaled(idx));
   'Next idx

   ' I write CAT, save, open, I see Inc instead
   For idx = 1 To sambufsize
     Inc writebufstr,Bin2str$(uint8,afcap_scaled(idx))
   Next idx
   Print #1; writebufstr
 EndIf

 If repeatcount = samrepeat Then Exit  ' Have all the samples been captured?
Loop

Print "recording complete."

Close #1
ADC close


'''''''''''''''''''''''''''''''''''''''
'sub pattern reference
 'sub mysub arg1, arg2$, arg3
 'end sub
'NOTE: No parameters can be passed to an interrupt handler subroutine

'''''''''''''''''''''''''''''''''''''''


''''''''''''''''''''''''''''''''''''''''
' Interrupt handling with flip-flop buffers for "ADC open <subname>"

'''''''''''''''''''''''''''''''''''''''
Sub _sam_ready()
 If repeatcount = samrepeat Then End  ' End if all samples are captured

 ' this is the only place where knowledge of the active buffer is needed
 If bufnum = 0 Then
     ' If bufnum 0 is full, start next capture into bufnum 1; scale for write
     ADC start afcap_1()
     Math scale afcap_0(),77,afcap_scaled()  ' 77 =~ 255/3.3 as integer
     bufnum = 1
 Else
     ' bufnum 1 is full, start capture into 0 and scale for write
     ADC start afcap_0()
     Math scale afcap_1(),77,afcap_scaled()  ' 77 =~ 255/3.3 as integer
     bufnum = 0
 EndIf

 samrunning = 1
 Inc repeatcount

 samready = 1
 samrunning = 0
End Sub
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 9122
Posted: 09:32pm 18 Nov 2023
Copy link to clipboard 
Print this post

Try this, seems to work perfectly for me

Option explicit
Option default integer
Const buffsize = 1000 '4000 samples
Const nsamples = buffsize * 4
Dim a(buffsize-1),b(buffsize-1),d(nsamples-1)

' print the wav header to the file here

ADC open 20000,1,myint 'one channel, sampled at 20000Hz"
Dim i,j,k,c=0, count=0
Dim s$
Open "B:/data.wav" For output As #1
ADC run a(),b()
Do
Loop Until c 'wait for the first conversion to complete
Do
   If c=1 Then
     Memory unpack a(),d(), nsamples, 16
   Else
     Memory unpack a(),d(), nsamples, 16
   EndIf
   Math scale d(),16,d() 'was 0-4095 scale to 0 to 65520
   i=0
   Do While i<buffsize
       s$=""
       j=Min(50,nsamples-i)
       For k=0 To j-1
           Inc s$,Bin2str$(uint16,d(i+k))
       Next
       Inc i,j
       Print #1,s$
       If j<>50 Then Print j
   Loop
   Inc count
   Print c;
   Timer =0
   c=0
   Do :Loop Until c
   Print Timer 'see how much time we have left before we need to process again
Loop Until count=100 ' arbitrary duration
Close #1
'
Sub myint
 c= MM.Info(ADC)
End Sub


OPTION SYSTEM SPI GP10,GP11,GP12
OPTION COLOURCODE ON
OPTION CPUSPEED  252000 'KHz
OPTION SDCARD GP15
OPTION AUDIO VS1053 GP2,GP3,GP4,GP5,GP6,GP7,GP8', ON PWM CHANNEL 1
 
Bleep
Guru

Joined: 09/01/2022
Location: United Kingdom
Posts: 509
Posted: 10:10pm 18 Nov 2023
Copy link to clipboard 
Print this post

I believe that second Memory unpack should be 'b()' I think?

  If c=1 Then
    Memory unpack a(),d(), nsamples, 16
  Else
    Memory unpack b(),d(), nsamples, 16
  EndIf

Edited 2023-11-19 08:11 by Bleep
 
leoec
Newbie

Joined: 13/11/2023
Location: United States
Posts: 6
Posted: 12:44am 19 Nov 2023
Copy link to clipboard 
Print this post

bleep: Agreed.  I noticed that.

matherp: That combination of: ADC run and the packed arrays should have a huge impact on performance.  I spent a little time scratching my head on syntax and other errors trying to run your code on 0707 and then decided to check 0708rc19's changelog and I see some of those features referenced.  I tried to upgrade and retest but I'm having some issue getting my sd card setup again in 0708.

Those new features look amazing for this kind of work.

I'll figure that out and let you know how it goes with the changes under 0708rc19.

Thank you.
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 9122
Posted: 01:49pm 19 Nov 2023
Copy link to clipboard 
Print this post

Please could you try this new version (RC20) and the new Basic approach and let me know results - thanks


PicoMite.zip


Option explicit
Option default integer
Const buffsize = 1000 '4000 samples
Const nsamples = buffsize * 4
const nbytes = buffsize * 8
Dim a(buffsize-1),b(buffsize-1)
ADC open 20000,1,myint 'one channel, sampled at 20000Hz"
Dim c=0, count=0
Dim s$
Open "B:/data.wav" For output As #1

' print the wav header to the file here

ADC run a(),b()
Do
Loop Until c 'wait for the first conversion to complete
Do
  If c=1 Then
     Math shift a(),4,a() ' shift all array elements 4 bits left =  multiply by 16
     Memory print #1,nbytes,a() 'print array direct to disk
  Else
     Math shift b(),4,b()
     Memory print #1,nbytes,b()
  EndIf
  Print c;
  Timer =0
  c=0
  Do :Loop Until c
  Print Timer 'see how much time we have left before we need to process again
  Inc count
Loop Until count=100 ' arbitrary duration
Close #1
'
Sub myint
c= MM.Info(ADC)
End Sub

Edited 2023-11-19 23:50 by matherp
 
leoec
Newbie

Joined: 13/11/2023
Location: United States
Posts: 6
Posted: 11:45pm 20 Nov 2023
Copy link to clipboard 
Print this post

Your new code runs straight away under rc20 without modification.  However, to play it I need the wave header or other visualization tools I haven't crafted.  I completed a thoughtful integration of your ADC routines into my existing adc2wav code and I get mostly noise.  It does bear some similarity in periodicity in common with the sampled radio signal.  I don't believe it is anything as simple as volume or an a()<->b() buffer reversal.  I didn't have enough time to dive deep enough to figure out what is going wrong.

I am preparing for and traveling through next weekend so this is about as far as I'll get until I have time after getting back.

My gut is that there is something wrong with the ADC run <array1>,<array2> output into those arrays.  Like they are aligned on an odd byte boundary or something.  Or maybe stored in big endian?  (IIRC WAVE is expecting little endian for the data).

Oh, also of note, I did comment out the x16 scaling shift (and play with different shifts) and sometimes had something closer to the audio source.  Finally, my original wave writing code worked for sure to write 8bit samples in WAV format.  I believe I modified it correctly to save 16 bit samples.  However, that is another undertainty.

When I get back my next step is probably to choose some very clean and predictable waveforms, like a 220Hz sine or square wave and look at the hexdump.  Not to mention bring a fresh perspective to it.

Just in case you or anyone else wants to experiment with the WAV output, my updated adc2wav.bas code is below:
' adc2wav v23 includes pmather ADC routines
' by Leo Edmiston-Cyr <leoec@gopher.quest>
' Last updated: 11/20/2023
'
Option explicit
Option default integer
Dim c=0, count=0

' print the wav header to the file here

Dim samdelay = 0          ' duration to delay before recording
Dim samduration = 10      ' duration of recording in seconds
Const sambufsize = 1000   ' 4000 16 bit samples; 12 bit resolution

Const sams_in_buf = sambufsize * 4    ' 4 16 bit samples per int
Const nbytes = sambufsize * 8         ' 8 bytes per int
Dim samrate = 20000                   ' samples per second (Hz)
Const samnum = samduration * samrate  ' total samples in recording
Const samrepeat = samnum / sams_in_buf  ' num sambufsize ADC cycles

Dim string filename = mm.cmdline$
If filename = "" Then
 filename = "adc2wav-" + Date$ + Str$(Epoch(now)) + ".wav"
EndIf

Dim samsize = 2 ' bytes per sample
Dim channum = 1 ' 1 = Mono; 2 = Stereo
Dim chunksize = 32 + samnum * samsize ' num samples *
Dim subChunkSize = 16
Dim byterate = samrate * channum * samsize
Dim sambits = samsize * 8 '8 bits per byte; obs
Dim afcap_0(sambufsize-1), afcap_1(sambufsize-1)

' Hi user, here's what's going on
Print "Recording to:",filename
Print "duration:",samduration
Print "channel(s):",channum
Print "bit depth:",samsize*8;"bits"
Print "sample rate:",samrate;"Hz"
Print "buffer size:",sambufsize;"byte(s)"
Print "buffer writes:",samrepeat
Print "total samples:",samnum

'
' write WAVE file format for later .wav playback
Open filename For output As #1

'The following metadata satisfies the RIFF WAVE format requirements
Print #1; "RIFF";
Print #1; Bin2str$(UINT32,chunksize);
Print #1; "WAVE";
Print #1; "fmt ";

' SubChunk
Print #1; Bin2str$(UINT32,subChunkSize);  ' the size of this sub chunkfor PCM
Print #1; Bin2str$(UINT16, 1); ' Audio format PCM = 1 is linear quantization
Print #1; Bin2str$(UINT16, channum); ' number of channels 2 = stereo
Print #1; Bin2str$(UINT32, samrate);  ' sample rate 8000, 44100, etc.
Print #1; Bin2str$(UINT32, byterate);
Print #1; Bin2str$(UINT16, channum * samsize); ' block align-bytes for one sample of all channels
Print #1; Bin2str$(UINT16, sambits);  ' bits per sample

' ExtraParams -- none for PCM encoding
'   ExtraParamSize -- doesn't exist for PCM
'   ExtraParams

Print #1; "data";  ' Subchunk2ID
' RIFF WAVE metadata is complete

' Initiate sampling of the audio input
'''''''''''''''''''''''''''''''''''''''
ADC open samrate,channum,myint
ADC run afcap_0(),afcap_1()

Do
Loop Until c 'wait for the first conversion to complete

Do
 If c=1 Then
    Math shift afcap_0(),4,afcap_0() ' shift all array elements 4 bits left =  multiply by 16
    Memory print #1,nbytes,afcap_0() 'print array direct to disk
 Else
    Math shift afcap_1(),4,afcap_1()
    Memory print #1,nbytes,afcap_1()
 EndIf
 'Print c;
 Timer =0
 c=0
 Do :Loop Until c
 'Print Timer 'time remaining before next sample is ready to write
 Inc count
Loop Until count=samrepeat ' arbitrary duration

Close #1
Print "recording complete."
ADC close

Sub myint
c= MM.Info(ADC)
End Sub
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 9122
Posted: 07:57am 22 Nov 2023
Copy link to clipboard 
Print this post

Best thing is to decompose this until you find my or your error.

Start with just the ADC RUN command and then unpack the array and plot it manually or using the new LINE PLOT command

Assuming that looks OK do the same again after first using the shift before the unpack.

Once that is all OK then you can look at the WAV header. NB: I think you are missing characters 41-44 in the header which certainly aren't optional when sending data to the VS1053
 
Print this page


To reply to this topic, you need to log in.

© JAQ Software 2024