meow5

1 It's pretty wild what I *don't* yet have in this 2 language: 3 4 * Basic math operations 5 * Conditionals 6 * Loops 7 * User-defined variables 8 9 The easiest one to rectify, I think, will be some basic 10 math operations. With that, I can at least have a decent 11 RPN calculator. 12 13 Before I do that, I'd like to now start separating the 14 language primitives I used in the creation of the 15 interpreter versus everyhing else. 16 17 Putting everything else in another file and including it 18 turns out to be super easy with NASM: 19 20 %include 'stdlib.asm' 21 22 So far, stdlib.asm contains just these: 23 24 ps (print stack) 25 inspect 26 inspect_all 27 all (all names) 28 29 So now I'll...YUCK, nevermind! NASM doesn't report the 30 line number of an error in an included file - it really 31 pretends the included content is actually in the file 32 that did the including. It was probably premature to 33 split this up anyway. 34 35 So everything will stay in meow5.asm as before. 36 37 Okay, so the math operations are easy because I'm just 38 popping the arguments, calling the CPU instructions, and 39 pushing the answer: 40 41 $ mr 42 20 8 - ps 43 12 44 4 * ps 45 48 46 3 / ps 47 0 16 48 "4 divided by 3 is $ remainder $\n" print 49 4 divided by 3 is 16 remainder 0 50 bin 51 1011 0100 + ps 52 1111 53 1111 hex ps 54 f f 55 0beef ps 56 f f beef 57 + ps 58 f befe 59 + ps 60 bf0d 61 Goodbye. 62 Exit status: 0 63 64 Increment and decrement! Bam! Easy! 65 66 45 inc ps 67 46 68 dec dec dec ps 69 43 70 71 I love easy stuff! 72 73 I'll need to figure out some more substantial stuff 74 next. Variables, maybe. Or conditionals. 75 76 Next week: I chose to implement variables first. Then I 77 took some time off to rest a bit. 78 79 Now I've got a first stab at 'var'. 80 81 (I originally planned to have variables put their VALUES 82 on the stack when called, but after I started 83 implementing them, I've decided to follow yet another 84 Forth convention: variables will put their ADDRESSES on 85 the stack. It's just so much simpler and more flexible 86 that way.) 87 88 So far, 'var' borrows from three existing words: 89 90 1. From 'colon', it takes the idea of getting the 91 next token from the input stream as a name. 92 93 2. From 'quote', it takes the compiling of the 94 machine code to push the immediate value 95 containing the address of the variable's memory. 96 97 3. From 'semicolon' - actually, it doesn't borrow, 98 it flat out *includes* semicolon. 99 100 Does it work? 101 102 var foo 103 hex foo "Address of foo: $\n" print 104 Address of foo: 804c16c 105 var bar 106 hex bar "Address of bar: $\n" print 107 Address of bar: 804c189 108 109 Seems plausible! So I guess I need new words to get 110 values to/from that address and the stack. 111 112 I think I'll call them "get" and "set" rather than the ! 113 ("store") and @ ("fetch") terminology from Forth. 114 115 They're easy to write in assembly. Just a handful of 116 instructions. 117 118 Here goes: 119 120 var foo 121 42 foo set 122 : foo? foo "Foo contains $\n" print ; 123 foo? 124 ./build.sh: line 34: 1465 Segmentation fault ./$F 125 Exit status: 139 126 127 Oops, I didn't even write that correctly. I forgot the 128 'get' in the definition of 'foo?'. But it should still 129 have printed the address of foo. So something has gone 130 awry. Now I know what I'm doing tomorrow night. 131 132 Next night: Okay, let's see where this crashes using 133 GDB. First, Can I set and get the variable? 134 135 Starting program: /home/dave/meow5/meow5 136 var foo 137 55 foo set 138 foo get 139 ps 140 134525204 55 141 142 Yes! Okay, I've got some extra garbage on the stack 143 (looks like an address that I'm not cleaing up at some 144 point). I'll deal with that in good time. 145 146 But 55 on the stack totally means 'foo get' worked. 147 Neat! 148 149 And can I print it from a string in immediate mode? 150 151 "foo is $\n" print 152 foo is 55 153 154 No problemo. 155 156 Now in a compiled word where it crashed last night. 157 (This time I remembered to do a 'get' as well): 158 159 : foo? foo get "foo=$\n" ; 160 foo? 161 162 Program received signal SIGSEGV, Segmentation fault. 163 0x0804b114 in token_buffer () 164 165 Yeah, so there we are. Apparently it crashes 166 while...what? Trying to execute code in token_buffer? 167 168 Well, I think the real problem is probably something 169 fundamental. Like, I haven't really thought through how 170 something works in a compiled word versus immediate 171 mode. 172 173 Next night: Okay, LOL, so first of all, I never 174 implemented numeric literals for compile mode. So I 175 can't even test strings like this: 176 177 $ mr 178 : x 5 "five is $" print ; 179 TODO: a new compile number word?Exit status: 0 180 181 And second of all, I don't think I ever tested number 182 interpolation (via '$' placeholders) in compiled words 183 either. Because that crashes: 184 185 $ mr 186 : x "stack has $" print ; 187 5 x 188 ./build.sh: line 34: 1390 Segmentation fault ./$F 189 Exit status: 139 190 191 So I should probably get those working. I'll start with 192 the second problem and work my way back to variable 193 printing: 194 195 [ ] Get $ placeholders working in compiled words 196 [ ] Implement compiled number literals 197 [ ] Implement compiled variable get/set 198 199 I'm not sure yet if that third item is even needed. 200 201 Okay, so I'm reviewing my 'quote' definition and the 202 thing that immediately stands out is that since I call 203 'quote' from the interpreter in immediate mode, it 204 should be "baking" in the placeholder value when the 205 string is put in memory. 206 207 So, for example: 208 209 12 210 211 : foo "I have $ coins." ; 212 213 6 214 215 foo 216 217 Should be printing "I have 12 coins." instead of "I have 6 218 coins." 219 220 Instead, it's crashing. 221 222 But before I fix that, I think I should re-think how 223 this works. Because I'm pretty sure as a programmer, I 224 would expect to have 'foo' output "I have 6 coins." 225 226 I mean, both ways have advantages and disadvantages. But 227 doing "late binding" on the placeholder seems much more 228 sensible and useful. I want the string to reflect the 229 value on the stack when I *run* 'foo', not when I define 230 it. 231 232 So I think what I need is for 'print' to contain the 233 placeholder stuff, not 'quote'! 234 235 Yuck, that's a pretty significant chnage. But I don't 236 really see a way around it if I want the language to 237 make any sense. 238 239 So, the new TODO is gonna have to be: 240 241 [ ] Make 'quote' just store the string with '$' 242 [ ] Make 'print' evaluate '$' placeholders in 243 strings (at "run time") 244 245 I guess this will also require a new output buffer so I 246 can build up strings to output. Or maybe I store them in 247 my general "free" memory as temporary storage? 248 249 Next night: Okay, let's get to this. 250 251 Step one is to not have 'quote' replace placeholder '$' 252 with numbers anymore. I'll just comment them out for 253 now. 254 255 Here's the simple example that crashed above: 256 257 : x "stack has $" print ; 258 5 x 259 stack has $ 260 261 Easy. 262 263 Next, 'print' needs to print a number from the stack 264 when it sees a '$' in the string. 265 266 There are two ways I could do this: 267 268 1. Make yet *another* copy of this string for 269 temporary printing purposes that has the number 270 string incorporated and print that. 271 272 2. Print the string up to the place holder, then 273 print the number, then print the next bit of 274 string after the placeholder, etc. 275 276 Hmmm... I'm not *loving* either of those options, The 277 disadvantage of #1 is that I need to find another place 278 to store the string - ideally temporary. The 279 disadvantage of #2 is that it's more complicated. 280 281 I feel like #1 can't be made "nicer". But maybe with 282 some thoughtful design, #2 could be broken into 283 palatable chunks of functionality. 284 285 Next night: Okay, after some painful stack and register 286 management, I've got it! 287 288 var foo 289 : foo? foo get "foo=$\n" print$ ; 290 55 foo set 291 foo? 292 foo=55 293 294 Okay, so I stuck with the plan to put the '$' 295 placeholder functionality in the printing mechanism 296 rather than the string building mechanism ('quote'). 297 298 Rather than turn 'print' into an abomination, I simply 299 added a 'print$' word that does number interpolation. 300 301 Here you can see 'print' vs 'print$' for comparison: 302 303 42 304 305 "--$--" print 306 --$-- 307 308 "--$--" print$ 309 --42-- 310 311 And 'print$' really wasn't too bad. I mean, it's still 312 quite compact. It was no fun to write. 313 314 I like how it cleaned up 'quote' a bit as well. That 315 word was way too long. 316 317 Okay, I need to add a *bunch* of stuff to my test script 318 next to double-check that what I have so far is 319 rock-solid. 320 321 I already know that something (semicolon?) is leaving an 322 address on the stack that it shouldn't. So I'll be 323 fixing that as well. 324 325 [ ] Fix whatever is leaving an addr on stack 326 327 Wait, I almost forgot the fun part: I was going to add a 328 new printing convenience word to wrap up 'print$' plus a 329 newline: 'say'. I love convenience! 330 331 1 2 + "I have $ coins." say 332 I have 3 coins. 333 334 Ah! I love it so much. Yes! What a difference one little 335 handy word makes. 336 337 I'm going update my hello world examples with 'say' now. 338 339 Done. 340 341 And now a confession: I've not been happy with my 342 Expect-based tests. Expect waits for the input you asked 343 for (until a timeout is reached) and it is perfectly okay with getting things you 344 *didn't* ask for. Which makes plenty of sense for 345 interacting with some applications. 346 347 But that does fit what I want to see, which is *exact* 348 output. And I want *immediate* failure when something 349 other than that is provided. 350 351 No doubt I can force Expect to do that, but I'm gonna 352 give shell scripts a shot. 353 354 First of all, I can't remember if I've tried piping 355 input into my interpreter yet, so here goes: 356 357 $ echo '"Hello" say' | ./meow5 358 Hello 359 Goodbye. 360 361 And we can make sure that I get my exact output with a 362 grep search: 363 364 $ echo '"Hello" say' | ./meow5 | grep '^Hello$' 365 Hello 366 $ echo $? 367 0 368 369 grep's 0 exit status means match was found. 370 371 $ echo '"Hello" say' | ./meow5 | grep '^foobar$' 372 $ echo $? 373 1 374 375 And 1 exit status is not found. 376 377 So scripting this is no problem. Here's my shell 378 function: 379 380 function try { 381 # -q tells grep to not echo, just return match status 382 if echo "$1" | ./meow5 | grep -q "^$2\$" 383 then 384 printf '.' 385 else 386 echo 387 echo "Error!" 388 echo "Wanted:" 389 echo "-------------------------------------------" 390 echo $2 391 echo "-------------------------------------------" 392 echo 393 echo "But got:" 394 echo "-------------------------------------------" 395 echo "$1" | ./meow5 396 echo "-------------------------------------------" 397 echo 398 fi 399 } 400 401 And some simple tests: 402 403 try '"Hello" say' 'Hello' 404 try '"Hello" say' 'beans' 405 406 $ ./test.sh 407 . 408 Error! 409 Wanted: 410 ------------------------------------------- 411 beans 412 ------------------------------------------- 413 414 But got: 415 ------------------------------------------- 416 Hello 417 Goodbye. 418 ------------------------------------------- 419 420 Cool, then I can re-write the Expect script with this 421 and verify the functionality so far. 422 423 Oh, and implement compiled numeric literals. That's 424 still an open TODO. 425 426 Next night: Cool as a consequence of the new test 427 script, I've fixed the address being left on the stack 428 (I had guessed 'semicolon', but it was actually 'colon'. 429 I was pushing the source address to copy the token 430 string for the new word/function name, but 'gettoken' 431 was already doing that, so it was redundant.) 432 433 I also implemented numeric literals in compile mode. I'm 434 getting a LOT of mileage out of that x86 opcode to push 435 an immediate 32 bit value onto the stack! 436 437 Now my tests look like: 438 439 # Input Expected Result 440 # ------------------- ------------------------ 441 try 'ps' '' 442 try '5 ps' '5 ' # note space 443 try '5 5 5 + ps' '5 10 ' 444 try '9 2 * ps' '18 ' 445 try '18 5 / ps' '3 3 ' 446 try '5 2 - ps' '3 ' 447 try '"Hello$\\\$\n" print' 'Hello$\\$' 448 try '"Hello" say' 'Hello' 449 try ': five 5 ; five ps' '5 ' 450 try ': m "M." print ; m '' say' 'M.' 451 try ': m "M." print ; : m5 m m m m m ; m5 '' say' 'M.M.M.M.M.' 452 453 And they all pass: 454 455 $ ./test.sh 456 ........... 457 Passed! 458 459 However, it looks like 'var' is pushing one to many 460 addresses because there's still one on my precious stack 461 that fouls up this test: 462 463 $ ./test.sh 464 ........... 465 Error! 466 Wanted: 467 ------------------------------------------- 468 var x 4 x set x get x ps 469 4 470 ------------------------------------------- 471 472 But got: 473 ------------------------------------------- 474 var x 4 x set x get x ps 475 134525172 4 134529332 476 Goodbye. 477 ------------------------------------------- 478 479 But I'm getting there! Feeling good about having these 480 tests to check against regressions as i add features. 481 482 I'll debug the stack issue with a bunch of DEBUG 483 statements to narrow it down like I did for 'colon'. 484 485 Next night: Yup, in fact it was the exact same bug I had 486 in 'colon'. Shame. 487 488 By the way, I found both of these by just peppering the 489 offending area my DEBUG macros and printing the value of 490 the esp register to see where something was being left 491 on the stack that shouldn't have been. It looked like 492 this once I fixed it: 493 494 var x 495 var start: ffafb9c0 496 var copys: ffafb9c0 497 var push: ffafb9bc 498 var end: ffafb9c0 499 4 x set 500 x get ps 501 4 502 x ps 503 4 134529376 504 get ps 505 4 4 506 507 Now I've fixed my test: 508 509 try 'var x 4 x set x get ps' '4 ' 510 511 And all is well: 512 513 $ mt 514 ............ 515 Passed! 516 517 And to finish it off, I need to test using (not 518 creating...yet) variables within compiled words. 519 520 The test: 521 522 var x 523 4 x set 524 : x? x get "x=$" say ; 525 x? 526 527 Expected result: 528 529 x=4 530 531 Crossing fingers: 532 533 $ mt 534 ............. 535 Passed! 536 537 Yay! 538 539 So it looks like that checks everything off for this 540 log: 541 542 [x] Get $ placeholders working in compiled words 543 [x] Implement compiled number literals 544 [x] Implement compiled variable get/set 545 [x] Make 'quote' just store the string with '$' 546 [x] Make 'print' evaluate '$' placeholders in 547 strings (at "run time") 548 [x] Fix whatever is leaving an addr on stack (twice) 549 550 See you in log10.txt!