Detokenising in 6502 machine code

BBC BASIC programs are tokenised, that is, BASIC keywords are stored as one-byte values This results in programs which execute faster and are more compact. BASIC has code to detokenise program code when listing, but it is unwise to try to call it directly as it is in different locations in different versions, and also depends on workspace locations containing values that BASIC expects which will not be the case if trying to use the routines when BASIC is no the current language.

The following code, which must run in the I/O processor, will search for the token table in any version of BBC BASIC - even in non-6502 BASIC ROMs.

   \ Detokenise BASIC tokens
   \ =======================
   \ needs, eg tbase=&70:tptr=&72 if calling from BASIC
   \        eg tbase=&A8:tptr=&AA if calling as a *command
   \
   \ ----------------------------------
   \ TokenInit - Find BASIC token table
   \ ----------------------------------
   \ On entry, BASIC ROM must be paged in
   \ On exit,  tbase=>Start of token table
   \           A,Y,tptr corrupted, X preserved
   \
   .TokenInit
   LDY #0:STY tptr:LDA #&80:STA tptr+1    :\ tptr=>ROM start
   .TokInitLp
   LDA #1:CLC:ADC tptr:STA tptr           :\ Inc. tptr
   LDA #0:TAY:ADC tptr+1:STA tptr+1
   LDA (tptr),Y:CMP #&80:BEQ TokInit2:DEY :\ Found &80
   .TokInit2
   INY:LDA (tptr),Y:CMP #ASC"A":BNE TokInitLp :\ Not "A"
   INY:LDA (tptr),Y:CMP #ASC"N":BNE TokInitLp :\ Not "AN"
   INY:LDA (tptr),Y:CMP #ASC"D":BNE TokInitLp :\ Not "AND"
   .TokInitOk
   LDA tptr:STA tbase:LDA tptr+1:STA tbase+1
   RTS
   :
   :
   \ -----------------------------
   \ PrToken - Print a BASIC token
   \ -----------------------------
   \ On entry, A=token byte
   \           BASIC ROM must be paged in
   \           tbase=>Start of token table, set by TokenInit
   \ On exit,  A,Y,tptr corrupted, X preserved
   \
   .TokenPrint
   PHA:LDA tbase:STA tptr           :\ Point to start token table
   LDA tbase+1:STA tptr+1
   .TokPrLp1
   LDY #&FF
   .TokPrLp2
   INY:LDA (tptr),Y:BPL TokPrLp2    :\ Loop until token byte found
   PLA:CMP (tptr),Y:BEQ TokPrFound  :\ Found matching token
   PHA:TYA:BNE TokPrStep            :\ Step to next token
   .TokPrLp3
   INY:LDA (tptr),Y:BPL TokPrLp3    :\ Find next token byte
   DEY:DEY
   .TokPrStep
   INY:TYA:SEC:ADC tptr:STA tptr    :\ Step past this token string
   LDA #0:ADC tptr+1:STA tptr+1
   BNE TokPrLp1                     :\ Loop to keep searching
   .TokPrFound
   TYA:BEQ TokPrNxt:LDY #0          :\ Skip past leading token
   .TokPrLp3
   LDA (tptr),Y:BMI TokPrEnd        :\ Token byte, end
   CMP #32:BCC TokPrEnd:JSR OSWRCH  :\ Flag byte, end
   .TokPrNxt
   INY:BNE TokPrLp3                 :\ Loop back for next character
   .TokPrEnd
   RTS

You can test the code with the following:

   CALL TokenInit
   FOR A%=&80 TO 255:PRINT ;A%;": ";:CALL TokenPrint:PRINT:NEXT

Note that the code requires the BASIC ROM to be paged in, and also needs to run in the I/O processor - where the ROMs are. This would normally be done in a *command utility with code such as the following:

   LDA &F4:PHA                      :\ Save current ROM
   LDA &24B:CMP #&FF:BEQ errNoBASIC :\ Get BASIC ROM, error if no BASIC
   AND #&127:JSR SelectROM          :\ Page in BASIC ROM
   ...
   JSR TokenInit
   ...
   JSR TokenPrint
   ...
   PLA                              :\ Restore previous ROM
   .SelectROM
   STA &F4:STA &FE30:RTS            :\ Page in ROM

This code has been tested with 6502 BASIC 1.00-4.00, 4.32, 4.40, 4.86, Z80 BASIC and PDP-11 BASIC.

See also

• Tokeniser

Jgharston 03:09, 22 November 2010 (UTC)