Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.
|
Forum Index : Microcontroller and PC projects : Fruitoftheshed giving me 404
Page 4 of 5 | |||||
Author | Message | ||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Well, wadaya know. I can retrieve the file using the Annex32 Basic WGET$ command with no problem with this code: a$=wget$("https://raw.githubusercontent.com/lyzby/FOTS-demo/main/MMBLibrary/ALPHANUM.BAS") wlog a$ PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Windows command line, GNU Wget 1.11.4. Thanks for checking. It worked where I wanted it to work--in Annex Basic. What I haven't yet succeeded at is getting the folder contents (listing file names in MMBLibrary). All the combinations I've tried return "404: Not Found" or "400: Invalid request". I am able to get and process that (i.e., break out the file names) with a WGET$ HTTP request for a local (windows) folder or remote file structure on a web site. EDIT: Ok, got it. Doing a WGET$ on the github URL ("https://github.com/lyzby/FOTS-demo/tree/main/MMBLibrary") returns a chunk of html code (761 lines) which includes lines like this (which can be parsed out): {"name":"main","listCacheKey":"v0:1705260040.0","canEdit":false,"refType":"branch","currentOid":"5d64515a636724e7eaa6a823e021ba9065b4a725"},"tree":{"items":[{"name":"1WDS1820.bas","path":"MMBLibrary/1WDS1820.bas","contentType":"file"},{"name":"ALPHANUM.BAS","path":"MMBLibrary/ALPHANUM.BAS","contentType":"file"},{"name":"ASTROFIX.BAS","path":"MMBLibrary/ASTROFIX.BAS","contentType":"file"},{"name":"AUTOLOG.bas","path":"MMBLibrary/AUTOLOG.bas","contentType":"file"},{"name":"AUTORUN.BAS","path":"MMBLibrary/AUTORUN.BAS","contentType":"file"},{"name":"Astrofix Readme.txt","path":"MMBLibrary/Astrofix Readme.txt","contentType":"file"},{"name":"BATTERY.BAS","path":"MMBLibrary/BATTERY.BAS","contentType":"file"}, Searching for {"name":"main" first finds "main", and all subsequent {"name" instances except for a few at the end identify file names. (That "main" is the repository "branch") So now, an Annex32 Basic program can, in theory, walk a github repository and retrieve the folder structure, file names, and files. I haven't yet tried to see how a subfolder is identified. ~ Edited 2024-01-15 08:54 by lizby PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
palcal Guru Joined: 12/10/2011 Location: AustraliaPosts: 1873 |
I'm obviously missing something and am probably about to learn something but why cant all The Fruit of the Shed files be kept here on TBS, there is already a Doc register. "It is better to be ignorant and ask a stupid question than to be plain Stupid and not ask at all" |
||||
JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 3802 |
Looks like a mismatch of SSL protocol (as it says), maybe you have an old version of something? Probably OpenSSL needs updating. John Edited 2024-01-15 09:49 by JohnS |
||||
phil99 Guru Joined: 11/02/2018 Location: AustraliaPosts: 2135 |
How does one access that? |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
From "Micromite MMBasic Library(s)", https://www.thebackshed.com/DocRegister/ But aside from that, I know nothing. PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
Turbo46 Guru Joined: 24/12/2017 Location: AustraliaPosts: 1611 |
The Doc Registry seems to have been removed from the menus for a while. It was restricted to only a few members for uploading but would have been a great place to store things like MMB4W and other things that are scattered around the forum. Bill Keep safe. Live long and prosper. |
||||
PeteCotton Guru Joined: 13/08/2020 Location: CanadaPosts: 368 |
Yeah - I remember those BLOBS in Oracle. They were pretty clunky. These are the new cloud Blobs. Rather than being a database object, it's more of a universal file store that can be downloaded via HTTP. All the cool kids are using them |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Can you provide links to an explanation and examples, and something about how you intend to use them for your project? PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
FWIW, here's code which runs on an ESP32-S2Mini flashed with Annex32 and retrieves the names of files in a github repository. Rough and [not] ready (with some partially tested code for retrieving files and folder lists from web pages). ' getFiles.bas crlf$=chr$(13)+chr$(10) qt$=chr$(34) source$="github" ' http dropbox onedrive blob url$="192.168.1.82/files/" url$="https://github.com/lyzby/FOTS-demo/tree/main/MMBLibrary" getType$="folder" serial2.mode 115200,16,18 ' tx,rx 'onserial2 getRequest gosub getRequest do loop getRequest: ' pause 500 ' plenty of time to receive entire request if getType$="folder" then sRequest$="GETDIR "+url$ sFiles$="" nFiles=0 gosub getFiles if sFiles$<>"" then wlog sFiles$+"# files: "+str$(nFiles) endif endif return getFiles: ' wlog sRequest$ if mid$(sRequest$,1,6)="GETDIR" then i=0 a$=wget$(mid$(sRequest$,8),80) ' wlog "Returned :"+a$ if source$="github" then j=instr(1,a$,"{""items""") if j>0 then j=instr(j,a$,"{""name""") jEnd=instr(j,a$,"}]") endif do while j>0 j2=instr(j+9,a$,"""") if j2>0 then ' file name found b$=mid$(a$,j+9,j2-j-9) sFiles$=sFiles$+b$+crlf$ nFiles=nFiles+1 ' gosub getObject j=instr(j2,a$,"{""name""") ' closing quote after file name if j>jEnd then ' past the end of the file names j=0 endif else j=0 ' something went wrong endif loop else if source$="http" then j=instr(1,a$,"<li>") do while j>0 j2=instr(j+4,a$,"</li>") if j2>0 then ' line found b$=mid$(a$,j+4,j2-j-4) gosub getObject j=instr(j2+5,a$,"<li>") else j=0 ' something went wrong endif loop endif endif return getObject: k=instr(1,a$,"'/files/") do while k>0 k=k+8 l=instr(k,a$,"'") s$=mid$(a$,k,l-k) if s$<>"../" then sFiles$=sFiles$+s$+crlf$ i=i+1 endif k=instr(l+1,a$,"'/files/") loop wlog str$(i)+" files"+crlf$+sFiles$ nFiles=i endif return oldgetFiles: i=0 a$=wget$("192.168.1.82/files/",80) k=instr(1,a$,"'/files/") do while k>0 k=k+8 l=instr(k,a$,"'") s$=mid$(a$,k,l-k) if s$<>"../" then sFiles$=sFiles$+s$+crlf$ i=i+1 endif k=instr(l+1,a$,"'/files/") loop wlog str$(i)+" files"+crlf$+sFiles$ nFiles=i return Partial (clipped) output: 1WDS1820.bas ALPHANUM.BAS ASTROFIX.BAS AUTOLOG.bas AUTORUN.BAS Astrofix Readme.txt BATTERY.BAS BATTERY.JPG BATTERY2.BAS ... Towers2.bas Towers2.fnt hexdump.bas julia.bas mndlbrot.bas printer2.bas # files: 79 PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
disco4now Guru Joined: 18/12/2014 Location: AustraliaPosts: 896 |
FotS Future - Update Gerry, Lance, do we have a plan going forward? Is either of you willing to be primary custodian of this Fruit of the Shed reborn? If you use a GitHub wiki I can provide some initial technical support, but I am making it up as I go along with the help of Google. Best wishes, Tom Tom, I have the zipped contents of the FotS. Its just under 40Meg and contains three directories. One for the pages, one for attachments and one for uploads. Those missing images appear to be links to images uploaded on TBS. Its too big to attach here, I have given a dropbox link to it below. I have (before Xmas) gone most of the way to writing a script (Purebasic program) to translate all the tags and links to the format understood by DokuWiki. It should not take much to modify it for GitHub wiki tags. In December I invested in a hosting service for 12 months so I could see how DokuWiki would work with the contents of FotS. My hesitation was around it being supported by one person not getting any younger. DokuWiki appears to be quite a commonly used wiki and available on most hosted services. My initial thoughts were to get the contents into Dokuwiki and see how it looked. The static copy of FOS will give a bit of breathing space. I have had a play with github wiki. The links below shows the Bacpkack Curver tracer article on GitHub wiki and on DokuWiki. As you can see the github wiki is capable of displaying the articles once converted. This article uses images from both an external site (TBS) and locally on GitGub wiki which can be displayed in line or as links. In summary github wiki could be used to hold the FotS and it would be free. My concerns are: FotS has 244 articles so 224 pages. The searching appears to only find words in the page title. There does not seem to be a way to implement the equivalent of namespaces i.e. categorisation, so you would always be looking at a long list of pages. For the repository to be cloned and work you need to use relative links. To add assets e.g. images to the wiki you need to add via the interface (drag drop or select) They get added and referenced as below. [Download PCB foil (solder side view)](https://github.com/disco4now/PicoMite/assets/49109003/bcc43a53-7b5f-45b7-b198-7ff24a173795) You can edit to make them relative and they work as both a link and inline. As a link: [Download PCB foil (solder side view)](../assets/49109003/bcc43a53-7b5f-45b7-b198-7ff24a173795) Inline: ![Download PCB foil (component side view)](../assets/49109003/df8abdd0-28ca-4d5b-ba01-e8229eaf1d67) You can do a bulk upload to a directory in the repository and reference the files as below. [Download PCB foil (solder side view)](https://github.com/disco4now/PicoMite/blob/main/assets/trackerfoilss.png) [Download PCB foil (component side view)](../blob/main/assets/trackerfoilcs.png) A relative link to an asset add this way does NOT seem to work for inline, i.e. ![Download PCB foil (component side view)](../blob/main/assets/trackerfoilcs.png) I do not see a way to bulk upload pages, so probably a cut and paste of prepared file would be required for each page. There is no granularity in permissions. The options as I understand are collaborators have full access, and then a choice of all others all being given either full access or read only. Curver tracer on GitHub wiki This is the article shown on the DokuWiki I am playing with. Curver tracer on Dokuwiki My preferred option is to move the FotS to DokuWiki. I have it hosted on VentraIP and would keep it going for as long as I am around. It gives good searching, full granularity for permissions and seems to be a well used/supported wiki. I can also bulk load the pages and supporting assets.I have broken the back of this and have just been pondering the rearrangement of the namespaces. I would see everyone having read access without a login and any TBS members in good standing. (i.e. a few posts and not kicked out) be given access on request to edit/ add pages. Multiple users can be made superuser to do admin if required. I have not raised with Glenn but as its PHP and runs on Linux and is a common application once is a proven thing maybe one day it could become https://www.thebackshed.com/wiki I am proposing to slighly rearrange the existing namespaces to this: The original MMBasic Library articles (78) (the zip file lizby is currently playing with) are moved to there own namespace. (they are early Maximite files mostly) CMM2 merged with MMBasic Hardware. Home Page MMBasic 105 (was 183) MMBasic Hardware 11 (was 4) MMBasic Original 78 (was 0) Circuit Ideas 9 Great Cow Basic 3 PIC ASM 7 Platform Agnostic 11 CMM2 0 (was 7) This link is to the 38MB file containing the FotS content. Contains: Attachments 67 in 46 folders 17MB Pages 224 in 7 folders 1.8MB Uploads 98 files in 5 folders 22MB fruitofthe shed content So for now I am working toward getting the data loaded into DokuWiki. Gerry Latest F4 Latest H7 |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
This is terrific, Gerry. You seem to be well along in the process. It all looks good to me. One question: does your arrangement include what is in Tom's github at MicroMite ArmMite and MMX Hardware This is a category Andrew set up (I think) but never added anything to. I put the Armmite F4 page there, and then, because I didn't see any place else to put them, the 37 Sensors and Picomite pages. I don't see that any MMX entries were ever made. (My stabs with MMBLibrary were just fumblings to try to get a little familiarity with github and programmatic access to it.) Lance PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
disco4now Guru Joined: 18/12/2014 Location: AustraliaPosts: 896 |
One question: does your arrangement include what is in Tom's github at MicroMite ArmMite and MMX Hardware Lance They will map into MMBasic Hardware I see at least 1 page like yours per MMBasic port in there. The LCD Panel Summary page will end up in there as well. Latest F4 Latest H7 |
||||
thwill Guru Joined: 16/09/2019 Location: United KingdomPosts: 4040 |
That's great Gerry. Given that you've made so much progress and I didn't want to be left holding the baby then I'm happy to leave it with you. Is there a simple way to export the raw wiki (not static HTML) from DokuWiki so that we can ensure someone can maintain an independent backup and thus have a higher truck number for this iteration of FoTS ? Lance, interesting process, but I believe you are ploughing your own furrow with regards retrieving files and lists from a GitHub repository. The canonical way to do that would be through the REST API (via curl on a desktop, HTTPClient on ESP32 - I think, I don't really know what I am talking about). That way I believe you only have to handle JSON output rather than scraping HTML. Best wishes, Tom Game*Mite, CMM2 Welcome Tape, Creaky old text adventures |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Thanks, Tom. I started scraping HTML from a local copy made from the FOTS HTTRACK file, which also worked for files I had access to on a web page of mine 2,000 miles away. This looked more generic than using the REST API, since it could also be extended for dropbox and onedrive file structures (I think). It appears that the portion of the github HTML file for MMBLibrary is in JSON format, where in ""tree":{"items":[{"name":"1WDS1820.bas"" the "[" indicates the beginning of an array in which each element (file name & attributes) is contained between "{" and "}". Unfortunately, the PicoMite does not include a JSON$ function. Unless more complete code is available I intend to use MikeO's JSONValue$ function: function JSONVALUE$(Jstring$,jParam$) ' by MikeO local a,c,s$,d$,b,json$ json$=jstring$ s$=":" d$="," 'locate parameter a=instr(JSON$,jparam$) 'print "A is:";a json$=replace$(json$,chr$(34),"") 'strip quotes json$=replace$(json$,"}",",") 'replace closing curly brackets with comma if a>0 then 'retrieve value 'look for separator : b=instr(a,json$,s$) if b>0 then 'look for deliminator c=instr(b,JSON$,d$) if c>0 then JSONVALUE$=mid$(JSON$,b+1,c-(b+1)) endif endif else JSONVALUE$="not found" endif end function Re the ESP32, I like Annex32 Basic on the ESP, and so far, its WGET$ command appears to do what is needed. (It also has JSON$.) Yes, I am ploughing my own furrow, since what I'm attempting is in line with Pete Cotton's idea, but intended to be applicable to more data sources, including wherever the FotS MMBLibrary ends up. ~ Edited 2024-01-15 23:47 by lizby PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
thwill Guru Joined: 16/09/2019 Location: United KingdomPosts: 4040 |
Ack, but note you will only see the "first" ~1000 files by scraping that HTML page (see warning message at top of https://github.com/thwill1000/thwill1000.github.io/tree/main/fruitoftheshed.com), I think you require the REST API to be able to page the file list properly. Best wishes, Tom Game*Mite, CMM2 Welcome Tape, Creaky old text adventures |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Thanks--but I was not looking to do this for any files other than those which would be program listings (like the MMBLibrary files). This method would not apply to the majority of FotS pages. An unstructured clump of more than 1,000 pages would probably not be very manageable or useful. ~ Edited 2024-01-16 00:23 by lizby PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
PeteCotton Guru Joined: 13/08/2020 Location: CanadaPosts: 368 |
Can you provide links to an explanation and examples, and something about how you intend to use them for your project? Sure. Every project/program is assigned a unique 5 character code (upper/lowercase and numeric). When the user enters the code, the ESP32 downloads the correpsonding index file <code>.index from the blob server. For example, for program HH56d it is https://controlsgeneratorblobs.blob.core.windows.net/mitenet/HH56d.index Which provides the following data (in text format) HH56d_a:\Heliblast\heliblast.bas,47800 HH56d_a:\Heliblast\gridarray.dat,63536 HH56d_a:\Heliblast\highscore.dat,800 HH56d_a:\Heliblast\assets\explosion.png,37311 HH56d_a:\Heliblast\assets\Drone2.png,294 HH56d_a:\Heliblast\assets\Title_8bit3.png,342493 HH56d_a:\Heliblast\assets\Blueprint1.png,108160 HH56d_a:\Heliblast\assets\stranger_-_run.mod,88210 HH56d_a:\Heliblast\assets\scrHighScore.bmp,768054 HH56d_a:\Heliblast\assets\scrPlayerSprites.bmp,768054 HH56d_a:\Heliblast\assets\scrEnemySprites.bmp,768054 HH56d_a:\Heliblast\assets\scrBlast.bmp,768054 HH56d_a:\Heliblast\assets\scrGroundSprites.bmp,768054 HH56d_a:\Heliblast\assets\scrUIElements.bmp,768054 Each of those files is part of the program (followed by the size in bytes). It is easy for the ESP32 to read through this file and build up the URL to download that blob, as follows. https://controlsgeneratorblobs.blob.core.windows.net/mitenet/HH56d.index]https://controlsgeneratorblobs.blob.core.windows.net/mitenet/<filename> I'm currently using a blob server from another of my projects for testing, but will create a dedicated server for the final result. So, some example links are: https://controlsgeneratorblobs.blob.core.windows.net/mitenet/HH56d_a:/Heliblast/heliblast.bas https://controlsgeneratorblobs.blob.core.windows.net/mitenet/HH56d_a:/Heliblast/assets/Title_8bit3.png Excellent. Thanks. |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Ok, thanks. I think I approximately understand. PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3150 |
Here is an Annex32 Basic program which retrieves the names of files from https://github.com/lyzby/FOTS-demo/tree/main/MMBLibrary using Annex's JSON$ function to extract the file names. ' getFilesJSON.bas dim itemName$(200) dim itemType$(200) crlf$=chr$(13)+chr$(10) source$="github" ' http dropbox onedrive blob url$="https://github.com/lyzby/FOTS-demo/tree/main/MMBLibrary" nFiles=0 gosub getFilesjson if nFiles>0 then for i=0 to nFiles-1 wlog str$(i+1)+" "+itemName$(i)+" "+itemType$(i) next i wlog crlf$+str$(nFiles)+" found" endif do loop getFilesjson: i=0 a$=wget$(url$,80) ' wlog "Returned :"+a$ if source$="github" then j=instr(1,a$,"""tree"":{""items""") ' find start of array of files jEnd=instr(j,a$,"}]") ' find end of array of files if j>0 and jEnd > j then c$="{"+mid$(a$,j+8,jEnd-j-6)+"}" ' wlog c$ ' wlog str$(j)+" "+str$(jEnd) for i=0 to 999 ' fname$=json$(c$,"items[i].name") fname$=json$(c$,"items.name["+str$(i)+"]") if fname$="not found" then exit for endif itemName$(nFiles)=fname$ ' fType$=json$(c$,"items[i].contentType") fType$=json$(c$,"items.contentType["+str$(i)+"]") if fType$="file" then itemType$(nFiles)="F" elseif fType$="not found" itemType$(nFiles)="?" else itemType$(nFiles)=ucase$(mid$(fType$,1,1)) endif nFiles=nFiles+1 if nFiles>199 then exit for endif next i endif return The chunk of html beginning "tree":{"items":[ and ending }] is already JSON code, so I substringed that out. Here @cicciocb (the author of Annex32) acknowledges that in several attempts at a canonical JSON$ function, he found it too resource-intensive, so used a method which requires a slightly different syntax. Instead of items[n].name, items.name[n] is required. I extracted the item names and types (all "File" in this case) and put them into 2 arrays. Here's a sample of the output: 1 ALPHANUM.BAS F 2 ASTROFIX.BAS F 3 AUTOLOG.bas F 4 AUTORUN.BAS F 5 Astrofix Readme.txt F 6 BATTERY.BAS F ... 74 Towers2.fnt F 75 hexdump.bas F 76 julia.bas F 77 mndlbrot.bas F 78 printer2.bas F 78 found The data in these arrays can be formatted in a way suitable to be sent to a 'mite (CMM2, PicoMite, F4) so that it could then retrieve a specific file through the ESP32. A similar method can be used on a website containing BASIC program files (and I assume with files from dropbox and onedrive (and I hope, DocuWiki)). ~ Edited 2024-01-16 09:59 by lizby PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
Page 4 of 5 |
Print this page |